Graph data storage method, graph calculation data conversion method and corresponding devices

Abstract

The invention relates to the technical field of graph data, and provides a graph data storage method, a graph calculation data conversion method and corresponding devices. The storage method comprises the steps that vertex data are stored by using a first file, the first file comprises a plurality of first-level blocks, and each first-level block corresponds to a horizontal division area in a vertex data table; storing the vertex data in the corresponding horizontal division region in the vertex data table into a corresponding first-level block of the first file in a column storage form; wherein the column storage mode specifically comprises the step of storing field data, belonging to the same field, in the vertex data in a continuous field storage space. According to the method and the device, the vertex data is stored in the memory by using the first file, so that the separation of the graph database and the graph calculation engine is realized, and the difficulty of reading the vertex data by the graph calculation engine is reduced by using a horizontal and vertical mixed storage mode.

Description

Technical Field

[0001] The present invention relates to the field of graph data technology, and in particular to a graph data storage method, a graph computing data conversion method and corresponding devices. Background Art

[0002] In the prior art, when the computing engine performs graph computing, the graph computing engine often directly reads the graph data from the graph database for computing. This implementation does not conform to the module separation principle of conventional system design, resulting in excessive coupling of the system. Once the graph computing engine or graph database changes, the coupling interface needs to be redesigned, which is obviously not conducive to later maintenance. In addition, due to the particularity of the data structure required for graph computing, the graph data in the graph database cannot be directly used for graph computing. Therefore, how to implement the technical architecture of computing separation (storage-computing), that is, to separate the operation logic of graph data storage and graph data computing, is a problem that technicians in this field need to solve.

[0003] In view of this, overcoming the defects of the prior art is an urgent problem to be solved in the field of this technology. Summary of the invention

[0004] The technical problem to be solved by the present invention is that in the prior art, the graph computing engine is directly coupled with the graph database, which is not conducive to subsequent maintenance.

[0005] The present invention adopts the following technical solution:

[0006] In a first aspect, the present invention provides a graph data storage method, using a first file to store vertex data, comprising:

[0007] The first file includes a plurality of first-level blocks, each first-level block corresponds to a horizontally divided area in the vertex data table;

[0008] The vertex data in the corresponding horizontally divided area in the vertex data table are stored in the corresponding primary block of the first file in the form of column storage; wherein the column storage method is specifically: field data belonging to the same field in each vertex data is stored in a continuous field storage space;

[0009] The first file also stores index information of each primary block, so that corresponding vertex data can be read from the first file according to the index information.

[0010] Preferably, storing the field data belonging to the same field in each vertex data in a continuous field storage space specifically includes:

[0011] Divide each field storage space into multiple secondary blocks;

[0012] storing a preset number of field data in each secondary block;

[0013] And the offset of each secondary block in the corresponding field storage space, the starting index value of the field data in each secondary block and the ending index value of the field data in each secondary block are written into the first file.

[0014] Preferably, the fields of each vertex data include original fields and newly added fields in the vertex data table; the original fields in the vertex data table include one or more of index value, physical ID, business ID and vertex type; the newly added fields include one or more of the number of adjacent in-points and the number of adjacent out-points;

[0015] The storing of a preset number of field data in each secondary block specifically includes:

[0016] Write the field data corresponding to the physical ID, service ID, number of adjacent entry points and number of adjacent exit points into the corresponding secondary block in sequence;

[0017] The field data corresponding to the index value and vertex type is written into the corresponding secondary block in the form of data compression.

[0018] Preferably, the step of writing the field data corresponding to the index value and the vertex type into the corresponding secondary block in a compressed form specifically includes:

[0019] Compressing all index values ​​in the secondary block into one piece of index compression data, wherein the index compression data includes a starting index value, an index increment, and an increment count;

[0020] Compress vertex type field data with adjacent positions and the same value in the secondary block into one vertex type compressed data; wherein the vertex type compressed data includes the value of the vertex type field data and the number of vertex type field data with the same value, and the vertex type field data is the field data corresponding to the vertex type.

[0021] Preferably, the method further comprises using a second file to store edge data; the fields of the edge data include one or more of an exit point, an entry point, a weight, and an edge type, specifically:

[0022] Write the field data corresponding to the outgoing point, the field data corresponding to the incoming point, and the field data corresponding to the edge type into the corresponding secondary block in a data compressed form;

[0023] Write the field data corresponding to the weight into the corresponding secondary block in order.

[0024] Preferably, writing the field data corresponding to the exit point, the field data corresponding to the entry point, and the field data corresponding to the edge type into the corresponding secondary block in a data compressed form specifically includes:

[0025] Compressing the out-point field data with adjacent positions and the same value in the secondary block into one out-point compressed data; wherein the out-point compressed data includes the value of the out-point field data and the number of out-point field data with the same value;

[0026] Compressing the entry point field data with adjacent positions and the same value in the secondary block into one entry point compressed data; wherein the entry point compressed data includes the value of the entry point field data and the number of entry point field data with the same value;

[0027] Compressing edge type field data that are adjacent in position and have the same value in the secondary block into one piece of edge type compressed data; wherein the edge type compressed data includes the value of the edge type field data and the number of edge type field data with the same value;

[0028] The exit point field data is the field data corresponding to the exit point, the entry point field data is the field data corresponding to the entry point, and the edge type field data is the field data corresponding to the edge type.

[0029] In a second aspect, the present invention provides a graph computing data conversion method, comprising:

[0030] Get graph data from the graph database;

[0031] Using the graph data storage method described in the first aspect, storing the vertex data in the graph data in a first file;

[0032] When performing graph calculation, the first file is read from the memory, vertex data is read from the corresponding first-level block according to the index information in the first file, and the data structure required for graph calculation is generated according to the vertex data.

[0033] Preferably, generating a data structure required for graph calculation according to the vertex data specifically includes:

[0034] The number of adjacent in-points and / or the number of adjacent out-points obtained by reading are accumulated in sequence and loaded into the array to obtain the corresponding offset array;

[0035] According to the index value of the vertex data read, the corresponding edge data is obtained in sequence, and the edge data is filled into the nbr array, edgeLables array and weights array to generate the data structure required for graph calculation.

[0036] In a third aspect, the present invention further provides a graph data storage device for implementing the graph data storage method described in the first aspect, the device comprising:

[0037] At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the processor to execute the graph data storage method described in the first aspect.

[0038] In a fourth aspect, the present invention further provides a graph computing data conversion device, which is used to implement the graph computing data conversion method described in the first aspect, and the device includes:

[0039] At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the processor to execute the graph computing data conversion method described in the second aspect.

[0040] In a fifth aspect, the present invention further provides a non-volatile computer storage medium, wherein the computer storage medium stores computer executable instructions, which are executed by one or more processors to complete the method described in the first aspect and / or the method described in the second aspect.

[0041] In a sixth aspect, a chip is provided, comprising: a processor and an interface, for calling and running a computer program stored in the memory from a memory, and executing the method as described in the first aspect to the second aspect and any one of them.

[0042] In a seventh aspect, a computer program product comprising instructions is provided, which, when executed on a computer or a processor, causes the computer or the processor to execute the method as in the first aspect to the second aspect and any one of the methods.

[0043] The present invention stores vertex data in memory by using a first file to store vertex data, thereby realizing separation of a graph database from a graph computing engine, and forms a horizontal and vertical mixed storage mode by dividing the primary block and field storage space, so that the graph computing engine can obtain specific fields of the vertex data after determining the primary block where the vertex data is located, thereby obtaining the required data without traversing the entire first file, thereby reducing the difficulty of the graph computing engine in reading vertex data. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments of the present invention. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0045] Figure 1 It is a schematic diagram of the process of a first method for storing graph data provided by an embodiment of the present invention;

[0046] Figure 2 It is a schematic diagram of a graph data storage method provided by an embodiment of the present invention;

[0047] Figure 3 is a schematic diagram of another method for storing graph data provided by an embodiment of the present invention;

[0048] Figure 4 It is a schematic diagram of the process of a second method for storing graph data provided by an embodiment of the present invention;

[0049] Figure 5 is a schematic diagram of another graph data storage method provided by an embodiment of the present invention;

[0050] Figure 6 It is a schematic diagram of the process of a third method for storing graph data provided by an embodiment of the present invention;

[0051] Figure 7 It is a schematic diagram of the process of a fourth method for storing graph data provided by an embodiment of the present invention;

[0052] Figure 8 is a schematic diagram of another method for storing graph data provided by an embodiment of the present invention;

[0053] Fig. 9 It is a flowchart of a graph computing data conversion method provided by an embodiment of the present invention;

[0054] Fig.10 It is a schematic diagram of a graph computing data conversion method provided by an embodiment of the present invention;

[0055] Fig.11 is a schematic diagram of another graph computing data conversion method provided by an embodiment of the present invention;

[0056] Fig.12 is a schematic diagram of another graph computing data conversion method provided by an embodiment of the present invention;

[0057] Fig.13 It is a schematic diagram of the architecture of a graph data storage device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0059] Unless the context requires otherwise, throughout the specification and claims, the term "including" is to be interpreted as open inclusion, that is, "including, but not limited to". In the description of the specification, the terms "one embodiment", "some embodiments", "exemplary embodiments", "examples", "specific examples" or "some examples" and the like are intended to indicate that specific features, structures, materials or characteristics associated with the embodiment or example are included in at least one embodiment or example of the present disclosure. The schematic representation of the above terms does not necessarily refer to the same embodiment or example. In addition, the specific features, structures, materials or characteristics may be included in any one or more embodiments or examples in any appropriate manner, that is, although they may be carried in the embodiments or examples of the above terms due to reasons such as the order and position of appearance, it is not limited to that they can be carried in combination by one embodiment or example.

[0060] In the description of the present invention, the terms "first" and "second" are used only for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present disclosure, unless otherwise specified, the meaning of "multiple" is two or more. In addition, for example, the same type of nouns may be described as two independent individuals by adding "A" and "B" at the end. In this case, the corresponding features defined as "A" and "B" are only used to distinguish the same type of individuals for description purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features.

[0061] In the description of the present invention, the expression "A and / or B" (where A and B are used to formally represent specific characteristic contents) will be involved, and the corresponding expressions include the following three combinations: only A, only B, and a combination of A and B.

[0062] As used herein, "about," "substantially," or "approximately" includes the stated value and an average value that is within an acceptable range of deviation from the particular value as determined by one of ordinary skill in the art taking into account the measurements in question and the errors associated with the measurement of the particular quantity (i.e., the limitations of the measurement system).

[0063] In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0064] Embodiment 1:

[0065] In order to realize the technical architecture of calculation separation, one feasible way is to store the graph data in the memory. A commonly used method in actual use is: writing each field data in the graph data into a file in the form of text and saving it, such as using csv text format to save it. However, this method occupies a large storage space on the one hand, and the text format needs to be converted in actual use, which makes it inconvenient to use. Moreover, when the graph computing engine only needs to load a certain vertex data, it needs to access each vertex data in the csv file from the beginning in sequence, resulting in low reading efficiency. In order to solve this problem, embodiment 1 of the present invention provides a graph data storage method, using a first file to store vertex data, such as Figure 1 As shown, including:

[0066] In step 201, the first file includes multiple first-level blocks, each of which corresponds to a horizontally divided area in the vertex data table; the graph data includes vertex data and edge data. In an optional embodiment, the edge data can be stored in the second file according to the same concept as the vertex data. In an actual application scenario, the first file is also called vertex.xgraphfmt, where xgraphfmt is a custom file format name, and the first-level block is also called RowBlock. The horizontally divided area can be understood as: dividing the vertex data table into multiple blocks according to rows, each block is a horizontally divided area, including multiple vertex data in a preset order, and the preset order is usually the order of index values ​​or the order of corresponding IDs. The horizontally divided area introduced here is intended to refer to: multiple vertex data arranged in sequence are stored in a first-level block.

[0067] In step 202, the vertex data in the corresponding horizontally divided area in the vertex data table are stored in the corresponding primary block of the first file in the form of column storage; wherein, the column storage method is specifically as follows: the field data belonging to the same field in each vertex data is stored in a continuous field storage space; and it is stored in the order of the size of the index value of the vertex data. It can be understood that: a vertex data includes multiple fields, a vertex data corresponds to a field data in each field, a field corresponds to a column in the vertex data table, and the index value is a field in the vertex data. Since when performing graph calculations, the reading of vertex data often does not obtain all its fields, but reads some of the fields therein to generate a corresponding array, so this embodiment stores the field data of the same field in a continuous field storage space to facilitate the reading of the graph calculation engine. In actual use, the field storage space is also called Column_Block.

[0068] In step 203, index information of each primary block is also stored in the first file, so that corresponding vertex data can be read from the first file according to the index information.

[0069] by Figure 2 For example, the vertex data includes six fields: index value, physical ID, business ID, vertex type, number of adjacent in-points and number of adjacent out-points. For example, the corresponding vertex data table can be as follows: Figure 2 As shown, the number of adjacent in-nodes and the number of adjacent out-nodes can be obtained by those skilled in the art from a graph database. Assuming that there are (k+1)N vertex data, the generated first file can be as follows: Figure 2 As shown, it includes index information and k first-level blocks, each of which stores N vertex data. Specifically: a first-level block includes m field storage spaces (m is the number of fields included in the vertex data, here m=6), and field data of the corresponding field is stored in each field storage space, such as storing field data corresponding to the index value in field storage space 1, and storing field data corresponding to the physical ID in field storage space 2, and so on. Field data corresponding to the number of adjacent points is stored in field storage space 2; wherein the field storage space in the first first-level block stores field data from the 1st to Nth rows, and the field storage space in the second first-level block stores field data from the N+1th to 2Nth rows, and so on. The field storage space in the kth first-level block stores field data from the kN+1th to (k+1)Nth rows. As shown Figure 2 As shown, in the second field storage space in the first primary block, the 1st to Nth row data of the physical ID are stored.

[0070] The index information can be understood as information used to determine the data reading position. Based on the different accuracy of the position to be determined by those skilled in the art, the index information can be implemented in a variety of ways. The following two implementations are taken as examples:

[0071] The first implementation method: the index information includes the starting address of each first-level block and the starting index value of the vertex data in each first-level block (that is, the minimum index value of all vertex data in the first-level block). Since each vertex data is stored in the order of the index value, it is possible to determine which first-level block the vertex data to be read is in based on the starting index value, and then read all vertex data from the first-level block based on the starting address of the first-level block, and then filter out the vertex data to be read. This implementation method is more suitable for the method of reading data as a whole in the first-level block.

[0072] The second implementation method: the index information includes the starting address of each field storage space and the starting index value of the vertex data in the first-level block, so that according to the starting index value, it is possible to determine which first-level block the vertex data to be read is in, and then determine the corresponding field storage space in the first-level block according to the field data to be read, so as to read the required field data from the field storage space. This implementation method is more suitable for the method of reading data in units of field data. In a specific implementation method, the index information includes: the starting address of each field storage space, the ending address of each field storage space, the starting index value of the vertex data in the first-level block, and the ending index value of the vertex data in the first-level block (that is, the maximum index value of all vertex data in the first-level block). Since in actual use, each field storage space is often continuous, the starting address of the next field storage space is usually the ending address of the previous field storage space plus one. Similarly, the starting address of the next first-level block is usually the ending address of the previous first-level block plus one, and the starting index value of the vertex data in the next first-level block is the ending index value of the vertex data in the previous first-level block plus one. The index information and each field data are stored in binary form in the first file. Figure 2 As shown, the index information is actually expressed as the block footer information and block footer index of each level-one block. The block footer information is also called RowFootMeta and is stored at the end of the corresponding level-one block. The block footer information of the corresponding level-one block includes the starting index value of the vertex data of the level-one block, the ending index value of the vertex data of the level-one block, the starting address of the storage space of each field in the level-one block, and the ending address of the storage space of each field in the level-one block. The block footer information of the corresponding level-one block is stored at the end of the level-one block. Figure 4As shown, the block tail index includes the starting address of each block tail information and the ending address of each block tail information and the total length of the area used to store the starting address of each block tail information and the ending address of each block tail information (in Figure 3 The starting address and the ending address of the block footer information are collectively referred to as RowFootMeta (Write_Start_Index, Write_End_Index). When reading vertex data, the preset byte size at the end of the first file is first read to obtain the Length, and then the starting address of the block footer index is reversed according to the Length, so as to read the starting address and the ending address of each block footer information in turn, and then locate the block footer information, and then determine which field storage space the field data to be read is located according to the block footer index, so as to read it.

[0073] It should be noted here that the horizontally divided area in the vertex data table corresponding to each first-level block is an example of the vertex data table in the graph database, and does not mean that the first file is associated with the graph database, or that the first file must be converted from the data in the graph database, but refers to the first-level block storing multiple vertex data.

[0074] This embodiment uses the first file to store vertex data, thereby storing the vertex data in the memory, thereby realizing the separation of the graph database and the graph computing engine, and by dividing the primary block and field storage space, a horizontal and vertical mixed storage mode is formed, so that the graph computing engine can obtain the specific field of the vertex data after determining the primary block where the vertex data is located, thereby obtaining the required data without traversing the entire first file, thereby reducing the difficulty of the graph computing engine in reading the vertex data.

[0075] In a preferred embodiment, the field data belonging to the same field in each vertex data is stored in a continuous field storage space, such as Figure 4 As shown, specifically including:

[0076] In step 301, each field storage space is divided into a plurality of secondary blocks; in actual use, the secondary blocks are also called PageBlocks.

[0077] In step 302, a preset number of field data is stored in each secondary block; the preset number is obtained by those skilled in the art based on demand analysis.

[0078] In step 303, the offset of each secondary block in the corresponding field storage space, the starting index value of the field data in each secondary block, and the ending index value of the field data in each secondary block are written in the first file. The offset of the secondary block in the corresponding field storage space is also called PageOffset. The starting address of the secondary block can be determined according to the offset and the starting address of the field storage space. The offset of the secondary block in the corresponding field storage space is stored at the end of the field storage space, and the starting index value of the field data in each secondary block and the ending index value of the field data in each secondary block are stored at the end of the secondary block.

[0079] by Figure 5 For example, for a field storage space, it is divided into j secondary blocks, and h pieces of field data are stored in each secondary block (where h×j=N). For example, for the second field storage space in the first primary block, it is divided into j secondary blocks, and the 1st to hth rows of data of the physical ID are stored in the first secondary block, and the h+1th to 2hth rows of data of the physical ID are stored in the second secondary block. And so on, the (j-1)h+1th to Nth rows of data of the physical ID are stored in the jth secondary block, and the starting index value and the ending index value of the field data (that is, the starting index value and the ending index value of the vertex data corresponding to the field data) are stored at the end of the corresponding secondary block. Figure 5 In the expression Page_block_p (min, max), p is greater than or equal to 1, and p is less than or equal to j), and the offset of each secondary block in the field storage space is stored at the end of the field storage space (in Figure 5 It is represented as PageOffset 1~j).

[0080] This embodiment further divides the field storage space to achieve multi-level stratification, which makes it easier for the graph computing engine to more accurately determine the storage location of the corresponding vertex data, reduce the number of queries or accesses, and further reduce the difficulty of data reading and improve the efficiency of data reading.

[0081] In an actual application scenario, the fields of each vertex data include the original fields in the vertex data table and the newly added fields; the original fields in the vertex data table include one or more of the index value, physical ID, business ID and vertex type; the newly added fields include one or more of the number of adjacent in-points and the number of adjacent out-points; the newly added fields are not originally in the vertex data table, but are newly added by those skilled in the art. For example, the fields of vertex data can be logically represented as<NO,RowID,PrimaryKey,LabelID,CNeighbor(IN),CNeighbor(OUT)> , where NO is the index value, and its data type in the first file is int32; RowID is the physical ID, and its field data in the first file is int64; PrimaryKey is the business ID, and its field data in the first file is string. For data of non-fixed length type, the length of the data is stored first, and then the data itself. For example, due to the non-fixed length of string type data, it can be stored in sequence.<len_1,str_1> ,<len_2,str_2> , …,<len_n,str_n> , where len_n is the length of the nth string type data, str_n is the nth string type data itself; LabelID is the vertex type, and the data type of the field data of this field in the first file is int32; CNeighbor(IN) is the number of adjacent in points, and the data type of the field data of this field in the first file is int32; CNeighbor(OUT) is the number of adjacent out points, and the data type of the field data of this field in the first file is int32. This embodiment adds fields for the number of adjacent in points and the number of adjacent out points to the graph data, so that the graph computing engine can directly read the field data of these two fields to obtain the number of adjacent points, which is convenient for allocating array space.

[0082] The preset number of field data is stored in each secondary block, such as Figure 6 As shown, specifically including:

[0083] In step 401, the field data corresponding to the physical ID, service ID, number of adjacent in points and number of adjacent out points are written into the corresponding secondary block in sequence; the sequence is the sequence of index values ​​in the point data.

[0084] In step 402, the field data corresponding to the index value and the vertex type are written into the corresponding secondary block in a data compressed form.

[0085] The field data corresponding to the index value and vertex type is written into the corresponding secondary block in the form of data compression, such as Figure 7 As shown, specifically including:

[0086] In step 501, all index values ​​in the secondary block are compressed into one piece of index compression data, wherein the index compression data includes a starting index value, an index increment, and an increase number.

[0087] Since the index value is an ascending self-increasing value of [0,N), the numerical representation of [0,N) is not directly written. Instead, the initial value (i.e., the starting index value), the increment (i.e., the index increment) and the number of increases are written. For example, assuming that the preset number of secondary blocks is 20,000, 0, 1, 19999 is written in the secondary block corresponding to the first index value, and 20000, 1, 19999 is written in the secondary block corresponding to the first index value, and so on. This can greatly reduce the amount of data required to be stored.

[0088] In step 502, the vertex type field data with adjacent positions and the same value in the secondary block are compressed into a vertex type compressed data; wherein the vertex type compressed data includes the value of the vertex type field data and the number of vertex type field data with the same value, and the vertex type field data is the field data corresponding to the vertex type. The adjacent positions refer to the adjacent storage positions, and the second vertex type field data in the secondary block is adjacent to the first vertex type field data position, and is also adjacent to the third vertex type field data position. For example, in a secondary block, there are seven vertex type field data, namely: 10001, 10001, 10001, 100001, 20001, 20001 and 20001, then these seven vertex type field data can be compressed into two vertex type compressed data, namely: <10001:4> and <20001:3>, where <10001:4> represents that there are 4 consecutive field data with the value of 10001, and <20001:3> represents that there are 3 consecutive field data with the value of 20001.

[0089] In actual use, graph data includes vertex data and edge data. For edge data, binary can be used directly for sequential storage. In a preferred embodiment, it can also be stored based on the same concept as the graph data, that is, the method also includes using a second file to store edge data; in actual use, edges are also called relationships, edge data are also called relationship data, and the second file is also called edge.xgraphfmt. It should be noted here that the storage method of the second file is the same as that of the first file, that is, it includes multiple first-level blocks, each of which includes multiple field storage spaces, each storage space includes multiple second-level blocks, and the fields of the edge data include one or more of the exit point, entry point, weight and edge type. Specifically: the field data corresponding to the exit point, the field data corresponding to the entry point and the field data corresponding to the edge type are written into the corresponding second-level block in the form of data compression; the field data corresponding to the weight is written into the corresponding second-level block in sequence. The various fields of the edge data can be logically represented as<Out,In,Weight,RelType> , where Out represents the exit point (specifically, the index value of the exit point), and the data type of the field data of this field in the first file is int32; In represents the entry point (specifically, the index value of the entry point), and the data type of the field data of this field in the first file is int32; Weight represents the weight, and the data type of the field data of this field in the first file is float; RelType represents the edge type, and the data type of the field data of this field in the first file is int32.

[0090] The compression method of the outgoing point field data, the incoming point field data and the edge type field data is implemented based on the same concept as the compression method of the vertex type field data in the graph data, that is, the field data corresponding to the outgoing point, the field data corresponding to the incoming point and the field data corresponding to the edge type are written into the corresponding secondary block in the form of data compression, specifically including:

[0091] The out-point field data with adjacent positions and the same value in the secondary block are compressed into one out-point compressed data; wherein the out-point compressed data includes the value of the out-point field data and the number of out-point field data with the same value.

[0092] The entry point field data with adjacent positions and the same value in the secondary block are compressed into one entry point compressed data; wherein the entry point compressed data includes the value of the entry point field data and the number of entry point field data with the same value.

[0093] The edge type field data with adjacent positions and the same value in the secondary block are compressed into one edge type compressed data; wherein the edge type compressed data includes the value of the edge type field data and the number of edge type field data with the same value.

[0094] The exit point field data is the field data corresponding to the exit point, the entry point field data is the field data corresponding to the entry point, and the edge type field data is the field data corresponding to the edge type.

[0095] In order to facilitate the reading of the first file and the second file, this embodiment also provides a preferred implementation method, that is, using a third file to store the number of vertices in the first file and the number of edges in the second file. In actual use, the third file is also called Meta.info, and other additional information can also be stored in the third file, such as the actual vertex type corresponding to each LableID, the actual edge type corresponding to each RelType, whether to load weights and other information.

[0096] The first file (vertex.xgraphfmt), the second file (edge.xgraphfmt) and the third file (Meta.info) are collectively referred to as the XGraphFormat storage format.

[0097] Embodiment 2:

[0098] The present invention is based on the method described in Example 1, combined with specific application scenarios, and uses technical descriptions in related scenarios to illustrate the implementation process of the present invention in characteristic scenarios.

[0099] This embodiment takes reading graph data from a graph database and storing the graph data in the XGraphFormat storage format as an example of an application scenario.

[0100] The graph data storage method described in this embodiment specifically includes:

[0101] Get graph data from the graph database, including vertex data, edge data, the number of vertex data and the number of edge data, store the vertex data in vertex.xgraphfmt, store the edge data in edge.xgraphfmt, and store the number of vertex data and the number of edge data in Meta.info.

[0102] In a specific application scenario, obtaining graph data from a graph database specifically includes:

[0103] The Cypher graph database query language is used to call methods to obtain all vertices in the graph from the graph database. After obtaining the vertex data, only its RowId, PrimaryKey, and LabelID are retained, and other attribute elements are removed. Here, RowId is of Long type, PrimaryKey is of string type, and LabelID represents the ID of the vertex label type. Since RowId is not a continuous self-increasing ID, it is necessary to number RowId (starting from 0 and increasing in sequence) to form<NO,RowID,PrimaryKey,LabelID> The four-tuple expression is:

[0104] MATCH(N)RETURN ID(N)AS ROWID,PK(N)AS PrimaryKey,LABEL(N)AS labelId

[0105] Generally speaking, a relationship represents the connection between vertices. A vertex may have one or more relationships. The opposite vertices are adjacent vertices, and these relationships are adjacent edges. According to the relationship flow, they can be divided into inbound and outbound directions. At the same time, if the relationship contains weight information that needs to be involved in the calculation, the weight value must be obtained at the same time. After obtaining all vertex data, traverse the adjacency relationships of all vertices. For example, traverse the vertices in the outbound direction of point A to obtain all adjacent vertices B, that is, execute the following statement:

[0106] MATCH(A)-[R]->(B)WHERE ID(A)=%S RETURN ID(A)AS V1,ID(B)AS V2,TYPE(R)AS RelType

[0107] Or, traverse the vertices in the outbound direction of A to obtain all adjacent vertices B and weight values ​​WEIGHT, that is, the following statement:

[0108] MATCH(A)-[R]->(B)WHERE ID(A)=%S RETURN ID(A)AS V1,ID(B)AS V2,R.WEIGHT AS WEIGHT,TYPE(R)AS RelType

[0109] Traverse the vertices in the direction of A to obtain all adjacent vertices C, that is, execute the following statement:

[0110] MATCH(C)-[R]->(A)WHERE ID(A)=%S RETURN ID(C)AS V1,ID(A)AS V2

[0111] Traverse the vertices in the outbound direction of A to obtain all adjacent vertices B and weight values ​​WEIGHT, that is, execute the following statement:

[0112] MATCH(C)-[R]->(A)WHERE ID(A)=%S RETURN ID(C)AS V1,ID(A)ASV2R.WEIGHT AS WEIGHT,TYPE(R)AS RelType

[0113] Record the peer relationship vertices of all vertices<SRC,DST,WEIGHT> , which can also be expressed as<Out,In,Weight> , and the number of adjacent vertices in the inbound and outbound directions<V,CNeighbor(IN),CNeighbor(OUT)>

[0114] Thus, the vertex data including 6 fields is obtained.<NO,RowID,PrimaryKey,LabelID,CNeighbor(IN),CNeighbor(OUT)> and edge data consisting of 4 fields<Out,In,Weight,RelType> .

[0115] This embodiment also describes the logical and physical organization format of XGraphFormat, and describes in detail the optimization process of this format in terms of storage overhead. To save all graph data, two types of xgraphfmt files, vertex.xgraphfmt and edge.xgraphfmt, need to be generated, followed by meta.info files.

[0116] vertex.xgraphfmt can be logically described by schema<NO,RowID,PrimaryKey,LabelID,CNeighbor(IN),CNeighbor(OUT)> They are respectively the vertex number (i.e. the index value in Example 1), physical ID, primary key (i.e. business ID), and label ID (also called vertex label, i.e. vertex type). The vertex number can determine the uniqueness of the vertex in the graph calculation process, RowID can determine the uniqueness of the vertex within the physical storage layer of the graph database, PrimaryKey (PK for short) can determine the uniqueness of the vertex at the data business level, and labelID determines the unique category of the vertex; CNeighbor (IN) represents the number of adjacent incoming nodes of the vertex, and CNeighbor (OUT) represents the number of adjacent outgoing nodes of the vertex.

[0117] edge.xgraphfmt can be logically described by schema as<SRC,DST,WEIGHT> They represent the vertex number of the starting point, the vertex number of the ending point, and the weight value of the relationship respectively.

[0118] Meta.info metadata mainly records the number of vertices (i.e. the number of vertex data) and the number of relationships (i.e. the number of edges) found in the query. For example, the format of data stored in Meta.info can be as follows:

[0119] {

[0120] "edgeNumber":1225, / / represents 1225 edge data

[0121] "vertexnumber":25566, / / represents 25566 vertex data

[0122] "LabelId":{ / / Vertex type and corresponding value

[0123] "Person":100001,

[0124] "Knows":200002,

[0125] },

[0126] "weighted":true / / represents loading weight

[0127] }

[0128] XGraphFormat files use a mixed row and column storage method to save the above data, such as Figure 8 As shown, in the process of writing data, horizontal and vertical segmentation are used to store data. Horizontal segmentation divides the entire data block according to the data capacity of each row. The data volume is horizontally segmented once every 2000w to form a RowBlock (i.e., a first-level block). Vertical segmentation is performed according to the number of columns described in the logical schema, and each column is a ColumnBlock (i.e., a field storage space). Therefore, in terms of logical representation, a RowBlock contains multiple ColumnBlocks, and from a horizontal view level, an XGraphFormat contains multiple RowBlocks. A ColumnBlock contains multiple PageBlocks (i.e., a second-level block) arranged in sequence. PageBlock is the smallest unit of logical storage, directly storing the vertex data and relationship data of XGraphFormat, and storing 2w rows of data in a PageBlock.

[0129] vertex.xgraphfmt contains 6 types of ColumnBlocks, namely NO_Column_Block (i.e., the first-level block for storing NO), RowId_Column_Block (i.e., the first-level block for storing RowID), PK_Column_Block (i.e., the first-level block for storing PrimaryKey), LabelId_Column_Block (i.e., the first-level block for storing LabelID), CNeighbor_IN_Column_Block (i.e., the first-level block for storing CNeighbor(IN)), and CNeighbor_OUT_Column_Block (i.e., the first-level block for storing CNeighbor(OUT)). The data types of these column blocks correspond to int32, int64, string, int32, int32, and int32, respectively. CNeighbor_IN_Column_Block and CNeighbor_OUT_Column_Block are used to allocate arrays directly when the graph computing engine reads data (based on the number of allocated arrays). If these two columns are not available, it is necessary to read all of them like a CSV file and then perform statistics on the adjacency of each point, which results in a lot of overhead. For data of non-fixed length, the length of the data is stored first, and then the data itself is stored. That is, for PK_Column_Block, the length of the field data is stored first (in Figure 8 len_m), and then store the field data itself (in Figure 8 In the representation as str_m), the length of the field data is stored in bytes of a fixed length. When reading, the length is read first, and then the field data of the subsequent length is read according to the length.

[0130] NO_Column_Block mainly stores the NO index of graph calculation. Since NO is an ascending self-increasing value of [0,N), the column is encoded using differential coding. Instead of directly writing the numerical representation of [0,N), a sequence with 0 as the initial value and 1 as the increment is written. The maximum-minimum value (Min-Max) of the PageBlock (i.e., the starting index value and the ending index value) can be recorded. Through the maximum-minimum value, it is possible to quickly determine whether the field data of the corresponding vertex data is located in the PageBlock, so there is no need to compare each No in turn. The specific page where the No is located can be directly found and read. Since 1 is the value of the same increment, it is sufficient to record the number of increments of 1. The value of PageBlock0 in NO_Column_Block0 can be directly represented by {0, delta = 1, repeat = 19999}. There are a total of 20,000 rows of graph calculation NO indexes, recording the value of Min-Max and the number of bytes occupied by the page. A NO_Column_Block contains {PageBlock0, PageBlock1...PageBlockN}. After a NO_Column_Block is written, the starting and offset position pointers (Write_Start_Index, Write_End_Index) of the column in the file are recorded. Write_Start_Index is the starting address, Write_End_Index is the ending address, and Write_End_Index = Write_Start_Index + PageBlock_Size.

[0131] Since RowId and PK data are not monotonic and unique, RowId_Column_Block and PK_Column_Block are written into the file in order of the original data values ​​without additional compression steps, and (Write_Start_Index, Write_End_Index) are recorded at the same time. Here, the Write_Start_Index pointer value should be the Write_End_Index+1 position of the last written Column_Block. The starting position information of ColumnBlock on the physical file is recorded here to facilitate quick positioning and reading.

[0132] In the process of writing PageBlock in LabelId_Column_Block, since the vertex labels may be the same, the LabelId may be continuous and consistent. In the process of writing files from memory, first record the current vertex LabelId, the number of consecutive repetitions is 0, and continue to read the next LabelId and record it as NextLabelId. If the two labels are inconsistent, then directly write the previous LabelId and read the next LabelId at the same time. If they are the same at this time, the number of consecutive repetitions is accumulated. This process continues until the next different labelId. First write the Repeat_Start byte identifier, then write the data {LabelId, repeat=N}, and finally write the Repeat_End byte identifier to obtain the vertex type compressed data. This method can realize data compression of the same LabelId locally. Record (Write_Start_Index, Write_End_Index) at the same time.

[0133] For example, suppose there are multiple consecutive vertex data whose LabelIds are:

[0134] Person0:100001

[0135] Person:1:100001

[0136] Person2:100001

[0137] Person3:100001

[0138] Software0:20001

[0139] Software1:20001

[0140] Software2:20001

[0141] It can be compressed into two vertex type compressed data:<Repeat_start:1001:4> and<Repeat_start:2001:3> The more continuous vertices there are, the higher the compression ratio is. When reading, you only need to read the labelId and then restore it according to the number of repeats.

[0142] CNeighbor_IN_Column_Block and CNeighbor_OUT_Column_Block represent the number of neighboring in-nodes and neighboring out-nodes. Since the number of neighbors in these two columns is too random, they are written into the file in the order of the original data values ​​without additional compression steps. (Write_Start_Index, Write_End_Index) is also recorded.

[0143] It should be noted here that, for the sake of simplicity of expression, this embodiment uses Write_Start_Index to represent the starting address, also called the starting position pointer, and uses Write_End_Index to represent the ending address, also called the offset position pointer, and represents the starting address and ending address of the block in the form of block (Write_Start_Index, Write_End_Index), where block can be a corresponding first-level block, a corresponding field storage space or a corresponding second-level block, such as using NO_Column_Block (Write_Start_Index, Write_End_Index) to represent the starting address and ending address of the field storage space corresponding to the index value.

[0144] This embodiment also uses Min to represent the starting index value, Max to represent the ending index value, and uses the form of block (Min, Max) to represent the starting index value and the ending index value of the block, such as NO_Row_Block (Min, Max) to represent the starting index value and the ending index value of the first-level block.

[0145] After all data in the current RowBlock is written, the block footer information RowFootMeta is recorded and written sequentially. RowFootMeta contains:

[0146] NO_Column_Block(Write_Start_Index,Write_End_Index)

[0147] RowId_Column_Block(Write_Start_Index,Write_End_Index)

[0148] PK_Column_Block(Write_Start_Index,Write_End_Index)

[0149] LabelId_Column_Block(Write_Start_Index, Write_End_Index)

[0150] CNeighbor_IN_Column_Block(Write_Start_Index,Write_End_Index)

[0151] CNeighbor_OUT_Column_Block(Write_Start_Index,Write_End_Index)

[0152] NO_Row_Block(Min,Max)

[0153] That is, write the start address and end address of each field storage space, as well as the start index value and end index value of the first-level block. The start index value and end index value of the first-level block are used to quickly locate the calculation No in different RowBlocks. For example, if the calculation No is located in RowBlock2, then RowBlock1 does not need to be accessed again. The start address and end address of CNeighbor_IN_Column_Block and the start address and end address of CNeighbor_OUT_Column_Block are used to implement cross-ColumnBlock row reading. To implement the reading of a vertex or relationship record, multiple pointers are required to locate the Write_Start_Index position of each ColumnBlock and then enter the PageBlock to read the data.

[0154] At the same time, the starting address and ending address RowFootMeta (Write_Start_Index, Write_End_Index) of the block tail information are recorded. RowFootMeta (Write_Start_Index, Write_End_Index) is recorded to the end of the file, and the total length of all RowFootMeta (Write_Start_Index, Write_End_Index) is stored in the preset bytes at the end of the file, that is, Figure 3 The Length in the file is used to determine the starting position of each RowFootMeta (Write_Start_Index, Write_End_Index) after reading the preset bytes at the end of the file to obtain the Length. After reading and obtaining RowFootMeta (Write_Start_Index, Write_End_Index), the block tail information can be read based on this. The position pointer of each Column_Block in the block tail information (that is, the starting address and ending address of each field storage space) can quickly locate the starting bit value and ending position of the Column_Block in random reading.

[0155] It also records the starting index value and ending index value NO_Page_Block (Min, Max) of each secondary block. NO_Page_Block (Min, Max) is used to determine whether a calculated NO exists in the current Page_Block; NO_Row_Block (Min, Max) can quickly determine whether the data of a calculated NO exists in the RowBlock. The priority is to first determine NO_Row_Block (Min, Max). If it is not in the range, jump to the next Row_Block. If it exists, traverse all Page_Blocks (Min, Max) in the NO_Column_Block of the current Row_Block until it finds which page block it is in.

[0156] When all Row_Blocks are written, the next Row_Block is written until all Row_Blocks are written into the vertex.xgraphfmt file. The above vertex.xgraphfmt file is generated.

[0157] This embodiment also describes the generation method of edge.xgraphfmt. edge.xgraphfmt contains 6 types of ColumnBlocks, namely Out_Column_Block (field storage space for storing outgoing points), In_Column_Block (field storage space for storing incoming points), Weight_Column_Block (field storage space for storing weights), and RelType_Column_Block (field storage space for storing edge types). They represent the calculation number of the outgoing point, the calculation number of the incoming point, the weight value on the edge, and the edge type Id. The data types of these column blocks correspond to int32, int32, float, and int32, respectively. Each record in the column block<Out,In,Weight,RelType> It can represent the only relationship in the graph database.

[0158] This embodiment obtains all the original vertex data through Cypher statements, converts RowId into calculation No, and obtains the original edge data (including weight) in the graph database through Cypher statements. Since in the process of traversing the edge, there will generally be a Source vertex (i.e., exit point) corresponding to multiple Target vertices (i.e., entry point), or a Target vertex corresponding to multiple Sorce vertices, the Out_Column_Block and In_Column_Block columns are compressed with the same vertex, and RelType_Column_Block also compresses the same edge type in the same way as the vertex type, and the edge data is written to PageBlock, ColumnBlock, and RowBlock in sequence.

[0159] For example, suppose there are multiple edge data source vertices and target vertices as shown in the following table:

[0160] Source vertex Target vertex 0 1 0 2 0 3 0 4 0 5 0 6

[0161] Since the entry points are the same, the compressed entry point data is<Out,0,6> , where 6 is the number of entry points with the same value.

[0162] Source vertex Target vertex 1 21 2 21 3 21 4 21 5 21 6 21

[0163] Since the output points are the same, the compressed output point data is<In,21,6> , where 6 is the number of outgoing points with the same value.

[0164] This compression method works best when there are super vertices in the graph. For example, when a vertex A in the graph has millions or more adjacent points, it can be compressed into 2 int32 bytes <0,99999999>.

[0165] This embodiment uses a custom XGraphFormat file format. This graph data storage format has a higher compression ratio than the data in the csv text format, and can better organize the graph data format. In a csv file, all fields are represented by text, and the actual type of each field cannot be specified, so it is difficult to compress the field on demand. However, xgraphformat can compress data according to field type to reduce storage space, and at the same time arrange and organize data in the form of ColumBlock and RowBlock, so as to achieve accurate positioning and reading of data.

[0166] Embodiment 3:

[0167] Based on Example 1 and Example 2, this embodiment also provides a graph computing data conversion method, such as Fig. 9 As shown, including:

[0168] In step 601, graph data is obtained from a graph database.

[0169] In step 602, the graph data storage method described in Example 1 is used to store vertex data in the graph data in a first file.

[0170] In step 603, when performing graph calculation, the first file is read from the memory, vertex data is read from the corresponding first-level block according to the index information in the first file, and the data structure required for graph calculation is generated according to the vertex data.

[0171] Among them, the method used to obtain graph data from the graph database has been described in detail in Example 2 and will not be repeated here.

[0172] The method of generating a data structure required for graph calculation based on the vertex data specifically includes: accumulating the number of adjacent incoming points and / or the number of adjacent outgoing points read in sequence, and loading them into an array to obtain a corresponding offset array; obtaining the corresponding edge data in sequence according to the index value of the read vertex data, and loading the edge data into the nbr array, edgeLables array and weights array to generate the data structure required for graph calculation.

[0173] The data structure required for the graph calculation is also called Compressed Sparse Data (CSD for short).

[0174] The offset array is a one-dimensional array in CSD, which mainly uses the accumulation method to store the number of neighbors of all points. The number of neighbors of the nth point is the value of the array subscript n minus the value of the array subscript n-1. Fig.10 For example, an offset array of size [0,3,5,6,8] is 5, which means that there are 5-1=4 points in the graph, the number of neighbors of the first point is 3-0=3, the number of neighbors of the second point is 5-3=2, the number of neighbors of the third point is 6-5=1, and the number of neighbors of the fourth point is 8-6=2. Depending on the offset array, the number of neighbors can be the number of adjacent out-points, the number of adjacent in-points, or the total number of adjacent out-points and adjacent in-points.

[0175] Therefore, by accumulating and arranging the neighboring points in the vertex data in sequence, the offset array related to the neighboring points can be obtained, that is, offset[i]=offset[i-1]+CNeighbor(IN)[i], where offset[i] is the i-th element value of the offset array, offset[i-1] is the i-1-th element value of the offset array, CNeighbor(IN)[i] is the number of neighboring points of the i-th vertex data, and offset[0]=0.

[0176] By accumulating and arranging the adjacent output points in the vertex data in sequence, we can get the offset array related to the adjacent output points, that is, offset[i]=offset[i-1]+CNeighbor(OUT)[i], where offset[i] is the i-th element value of the offset array, offset[i-1] is the i-1-th element value of the offset array, CNeighbor(OUT)[i] is the number of adjacent output points of the i-th vertex data, and offset[0]=0.

[0177] By sequentially accumulating and arranging the adjacent input points and adjacent output points in the vertex data, an offset array related to the adjacent input points and adjacent output points can be obtained, that is, offset[i]=offset[i-1]+CNeighbor(IN)[i]+CNeighbor(OUT)[i], wherein offset[i] is the i-th element value of the offset array, offset[i-1] is the i-1th element value of the offset array, CNeighbor(IN)[i] is the number of adjacent input points of the i-th vertex data, CNeighbor(OUT)[i] is the number of adjacent output points of the i-th vertex data, and offset[0]=0. It can be seen that under the graph data storage method of Example 1, when constructing the offset array, it is only necessary to read the field storage space corresponding to the number of adjacent input points and / or the number of adjacent output points, without reading other fields, thereby greatly improving the construction efficiency of the offset array.

[0178] After the offset array is constructed, initialize CSD, pass in the constructed offset array to initialize CSD, initialize the total number of edges, whether to load weights, whether to load edge labels, data model, compression method, etc. The most important thing is to pass in offset[vertexCount], that is, the total number of edges, to apply for space for the nbr array. nbr is a one-dimensional array in CSD, which is used to store all adjacencies of the entire graph. When calculating, obtaining the adjacency of nbr points requires the use of the offset array. For example, there are 4 vertices in a graph, 1, 2, 3, and 4. [0,3,5,6,8] is the offset array, and [2, 3, 4, 1, 3, 1, 1, 2] is the corresponding nbr array. Then the adjacency of point 1 is 2, 3, 4, the adjacency of point 2 is 1, 3, the adjacency of point 3 is 1, and the adjacency of point 4 is 1, 2. If the calculation requires weights, initialize the weights array. If the calculation requires edge labels, initialize the edgeLabels array. The elements corresponding to the subscripts of the above two arrays correspond one-to-one to the edges in the nbr array. That is, the same subscript represents an adjacency of a point and the weight and edge label of the adjacency. The corresponding relationship is as follows: Fig.10 shown.

[0179] CSD is loaded, nbr is loaded, and all edge data and attributes are recorded in the XGraphFormat file. When reading, the corresponding adjacent data is read according to the direction (both, in, out) and loaded into the nbr array. At the same time, the weights array and edgeLabels array are loaded according to whether weights and relationships are needed. The advantage is that only one array is used to store the edges of the entire graph, which greatly reduces memory usage.

[0180] This embodiment also combines the application scenarios of distributed graph computing and uses technical descriptions in related scenarios to illustrate the implementation process in the characteristic scenarios of the present invention.

[0181] The graph computing data conversion method described in this embodiment also includes: reading the number of vertex data from a third file, and determining the number of points that each computing node needs to load based on the number of vertex data. Specifically: In the XGraphFormat file, the number of all points in the graph is recorded. When loading data, this value is read and the points that each computing node should load are calculated. For example: there are 10,000 points in the graph, and 3 computing nodes are used for calculation. By using the point averaging method, the point id interval that each computing node is responsible for calculating can be obtained. The first computing node calculates points in the interval [0,3333], the second computing node calculates points in the interval [3334,6667], and the third computing node calculates points in the interval [6668,9999]. The averaging result is as follows: Fig.11 As shown, Fig.11 shown.

[0182] Point data segmentation reading, because file reading is sequential reading, it is necessary to configure filtering conditions when reading XGraphFormat files to filter out data that does not belong to the computing node. This operation can greatly improve the system reading efficiency. Configure the target column No in the calculation interval assigned to the corresponding computing node. The read data is all required by the computing node, that is, by calculating No to find (Min, Max), each computing node reads the data belonging to its own No partition in xgraphformat, without having to read all of it.

[0183] Edge data segmentation reading, because all edges are put together, and when they are put in, they are not segmented according to the corresponding intervals of the points, so each computing node should read all the edge data, and judge whether the edge belongs to the computing node according to the direction and point during the reading process. For example: there are 10,000 points in the figure, and 3 computing nodes are calculating. The first computing node calculates the points in the interval [0,3333], the second computing node calculates the points in the interval [3334,6667], and the third computing node calculates [6668,9999] There are now two edges in the interval, edge 1 <exit point: 1, entry point: 7000, weight: 2.0, edge label: knows>, edge 2 <exit point: 5000, entry point: 3, weight: 1.2, edge label: knows>. Load in the in direction. The entry point of edge 1 is 7000, which belongs to the interval [6668,9999]. So edge 1 should be written by the third computing node. The entry point of edge 2 is 3, which belongs to the interval [0,3333]. So edge 2 should be written by the first computing node. Load in the out direction. The exit point of edge 1 is 1, which belongs to the interval [0,3333]. So edge 1 should be written by the first computing node. The exit point of edge 2 is 5000, which belongs to the interval [3334, 6667]. So edge 2 should be written by the second computing node. If loaded in both directions, the first computing node's incoming adjacency array will be written into edge 2, and its outgoing adjacency array will be written into edge 1. The second computing node's outgoing adjacency array will be written into edge 2, and the third computing node's incoming adjacency array will be written into edge 1. The partitioning result is as follows: Fig.12 shown.

[0184] Embodiment 4:

[0185] On the basis of Embodiment 1 and Embodiment 2, this embodiment further provides a graph data storage device, including:

[0186] At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the processor to execute the graph data storage method described in any one of Examples 1-2.

[0187] Based on Example 3, this embodiment further provides a graph computing data conversion device, including:

[0188] At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the processor to execute the graph computing data conversion method described in Example 3.

[0189] The graph computing data conversion device and the graph data storage device are implemented based on the same concept. This embodiment only takes the graph data storage device as an example. Fig.13 FIG. 2 is a schematic diagram of the architecture of a graph data storage device according to an embodiment of the present invention. The graph data storage device according to this embodiment includes one or more processors 21 and a memory 22. Fig.13 A processor 21 is taken as an example.

[0190] The processor 21 and the memory 22 may be connected via a bus or other means. Fig.13 The example of connecting through bus is taken in the following.

[0191] The memory 22 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs and non-volatile computer executable programs, such as the graph data storage method in Example 1. The processor 21 executes the graph data storage method by running the non-volatile software programs and instructions stored in the memory 22.

[0192] The memory 22 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage devices. In some embodiments, the memory 22 may optionally include a memory remotely arranged relative to the processor 21, and these remote memories may be connected to the processor 21 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0193] The program instructions / modules are stored in the memory 22, and when executed by the one or more processors 21, the graph data storage method in the above-mentioned embodiment 1 is executed.

[0194] It is worth noting that the information interaction, execution process, etc. between the modules and units within the above-mentioned devices and systems are based on the same concept as the processing method embodiment of the present invention. The specific contents can be found in the description of the method embodiment of the present invention and will not be repeated here.

[0195] A person skilled in the art may understand that all or part of the steps in the various methods of the embodiments may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the storage medium may include: a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc.

[0196] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for storing graph data, characterized in that: Use the first file to store vertex data, including: The first file includes a plurality of first-level blocks, each first-level block corresponds to a horizontally divided area in the vertex data table; The vertex data in the corresponding horizontally divided area in the vertex data table are stored in the corresponding primary block of the first file in the form of column storage; wherein the column storage method is specifically: field data belonging to the same field in each vertex data is stored in a continuous field storage space; The first file also stores index information of each primary block, so that corresponding vertex data can be read from the first file according to the index information.

2. The method for storing graph data according to claim 1, characterized in that: The storing of the field data belonging to the same field in each vertex data in a continuous field storage space specifically includes: Divide each field storage space into multiple secondary blocks; storing a preset number of field data in each secondary block; And the offset of each secondary block in the corresponding field storage space, the starting index value of the field data in each secondary block and the ending index value of the field data in each secondary block are written into the first file.

3. The method for storing graph data according to claim 2, characterized in that: The fields of each vertex data include original fields and newly added fields in the vertex data table; the original fields in the vertex data table include one or more of index value, physical ID, business ID and vertex type; the newly added fields include one or more of the number of adjacent in-points and the number of adjacent out-points; The storing of a preset number of field data in each secondary block specifically includes: Write the field data corresponding to the physical ID, service ID, number of adjacent entry points and number of adjacent exit points into the corresponding secondary block in sequence; The field data corresponding to the index value and vertex type is written into the corresponding secondary block in the form of data compression.

4. The method for storing graph data according to claim 3, characterized in that: The step of writing the field data corresponding to the index value and the vertex type into the corresponding secondary block in a data compressed form specifically includes: Compressing all index values ​​in the secondary block into one piece of index compression data, wherein the index compression data includes a starting index value, an index increment, and an increment count; Compress vertex type field data with adjacent positions and the same value in the secondary block into one vertex type compressed data; wherein the vertex type compressed data includes the value of the vertex type field data and the number of vertex type field data with the same value, and the vertex type field data is the field data corresponding to the vertex type.

5. The method for storing graph data according to claim 1, characterized in that: The method further includes using a second file to store edge data; the field of the edge data includes one or more of an exit point, an entry point, a weight, and an edge type, specifically: Write the field data corresponding to the outgoing point, the field data corresponding to the incoming point, and the field data corresponding to the edge type into the corresponding secondary block in a data compressed form; Write the field data corresponding to the weight into the corresponding secondary block in order.

6. The method for storing graph data according to claim 5, characterized in that: The step of writing the field data corresponding to the exit point, the field data corresponding to the entry point, and the field data corresponding to the edge type into the corresponding secondary block in a data compressed form specifically includes: Compressing the out-point field data with adjacent positions and the same value in the secondary block into one out-point compressed data; wherein the out-point compressed data includes the value of the out-point field data and the number of out-point field data with the same value; Compressing the adjacent entry point field data with the same value in the secondary block into one entry point compressed data; wherein the entry point compressed data includes the value of the entry point field data and the number of entry point field data with the same value; Compressing edge type field data that are adjacent in position and have the same value in the secondary block into one piece of edge type compressed data; wherein the edge type compressed data includes the value of the edge type field data and the number of edge type field data with the same value; The exit point field data is the field data corresponding to the exit point, the entry point field data is the field data corresponding to the entry point, and the edge type field data is the field data corresponding to the edge type.

7. A graph data storage device, characterized in that: include: at least one processor; And, a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the processor to execute the graph data storage method described in any one of claims 1-6.

8. A graph computing data conversion method, characterized in that: include: Get graph data from the graph database; Using the graph data storage method according to any one of claims 1 to 6, storing the vertex data in the graph data in a first file; When performing graph calculation, the first file is read from the memory, vertex data is read from the corresponding first-level block according to the index information in the first file, and the data structure required for graph calculation is generated according to the vertex data.

9. The graph computing data conversion method according to claim 8, characterized in that: Generating the data structure required for graph calculation according to the vertex data specifically includes: The number of adjacent in-points and / or the number of adjacent out-points obtained by reading are accumulated in sequence and loaded into the array to obtain the corresponding offset array; According to the index value of the vertex data read, the corresponding edge data is obtained in sequence, and the edge data is filled into the nbr array, edgeLables array and weights array to generate the data structure required for graph calculation.

10. A graph computing data conversion device, characterized in that: include: at least one processor; And, a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the processor to execute the graph computing data conversion method described in claim 8 or 9.

Images