Key-value pair storage method for graph data and graph data prefetching method

Abstract

One or more embodiments of the present specification provide a key value pair storage method and a graph data prefetching method for graph data, the method comprising: obtaining a plurality of graph composition element key value pairs, in any one of the graph composition element key value pairs: the value thereof corresponds to the graph composition element data of the corresponding graph composition element in the graph data, the key comprises a plurality of identification fields, and one of the identification fields is a category identifier of the corresponding composition element; wherein the composition elements corresponding to the plurality of composition element key value pairs comprise nodes and / or edges; and orderly storing the composition element key value pairs in the plurality of composition element key value pairs according to an arrangement sequence among the category identifiers contained in the corresponding keys.

Description

Technical Field

[0001] One or more embodiments of the present specification relate to the field of graph data, and in particular, to a key-value pair storage method for graph data and a graph data pre-fetching method. Background Art

[0002] Graph data is a data organization model based on the graph structure in graph theory. Because it can represent various entities and their relationships in the real world, it has obvious advantages in processing highly interconnected data and is widely used in knowledge graphs, social network analysis, financial services and other fields.

[0003] However, the scale of graph data is constantly expanding, and the complexity of analysis requirements is also increasing. As a result, related systems need to read and process large amounts of data during graph data-based analysis, which in turn triggers high-frequency hard disk read operations, affecting overall performance. Summary of the invention

[0004] In view of this, one or more embodiments of the present specification provide a key-value pair storage method for graph data and a graph data pre-fetching method.

[0005] To achieve the above objectives, one or more embodiments of this specification provide the following technical solutions:

[0006] According to a first aspect of one or more embodiments of this specification, a key-value pair storage method for graph data is proposed, including:

[0007] Acquire multiple composition element key-value pairs, in which: in any composition element key-value pair: its value corresponds to the composition element data of the corresponding composition element in the graph data, its key includes multiple identification fields and one of the identification fields is the category identification of the corresponding composition element; wherein the composition elements corresponding to the multiple composition element key-value pairs include nodes and / or edges;

[0008] The composition element key-value pairs in the plurality of composition element key-value pairs are stored in order according to the arrangement order between the category identifiers contained in the corresponding keys.

[0009] According to a second aspect of one or more embodiments of this specification, a graph data pre-fetching method is proposed, including:

[0010] Receive a graph data reading request for a graph database; the graph database stores a plurality of composition element key-value pairs, each composition element key-value pair is stored in order according to the arrangement order between the category identifiers contained in the corresponding key, and for any composition element key-value pair: its value corresponds to the composition element data of the corresponding composition element in the graph data, its key contains a plurality of identification fields and one of the identification fields is the category identifier of the corresponding composition element; wherein the composition elements corresponding to the plurality of composition element key-value pairs include nodes and / or edges;

[0011] Determine a matching key and a storage address of a corresponding composition element key-value pair from the plurality of composition element key-value pairs;

[0012] For a pre-fetch address range that matches the determined storage address of the composition element key-value pair, the composition element key-value pairs stored in the pre-fetch address range are read.

[0013] According to a third aspect of one or more embodiments of this specification, an electronic device is proposed, comprising: a processor; a memory for storing processor executable instructions; wherein the processor implements the steps of the method described in the first aspect or the second aspect above by running the executable instructions.

[0014] According to a fourth aspect of one or more embodiments of this specification, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method described in the first aspect or the second aspect are implemented.

[0015] According to a fifth aspect of one or more embodiments of this specification, a computer program product is proposed, including a computer program / instruction, which implements the method described in the first aspect or the second aspect when executed by a processor.

[0016] In the technical solution provided in this specification, when the data of the composition elements in the figure are stored in the form of key-value pairs, the key form containing multiple identification fields in each composition element key-value pair is organized, and the category identifier is used as one of the identification fields. Then, the key-value pairs of each composition element are stored in order according to the arrangement order between the category identifiers contained in the multiple keys. Applying the technical solution of this specification, the data of the composition elements in the figure can be stored in a centralized and orderly manner, ensuring that the key-value pairs of composition elements of the same category can be continuously stored in adjacent physical locations in the storage medium. While reading the target data, if the data stored in the adjacent physical position is pre-read, the correlation between the pre-read data and the target data can be improved. Therefore, after the pre-read data is cached, the hit rate of the subsequent cached data can be improved, avoiding high-frequency hard disk reading operations, thereby greatly improving the reading efficiency of the image data.

[0017] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the embodiments of the present specification. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is a schematic diagram of an overall architecture provided by an exemplary embodiment.

[0019] Figure 2 It is a flowchart of a key-value pair storage method for graph data provided by an exemplary embodiment.

[0020] Figure 3 It is a schematic diagram of a graph structure provided by an exemplary embodiment.

[0021] Figure 4 It is a schematic diagram of a key including a composition element type identifier provided by an exemplary embodiment.

[0022] Figure 5 It is a schematic diagram of a key including a time window identifier provided by an exemplary embodiment.

[0023] Figure 6 is a schematic diagram of a key including a placeholder provided by an exemplary embodiment.

[0024] Figure 7 It is a flowchart of a graph data pre-fetching method provided by an exemplary embodiment.

[0025] Figure 8 It is a structural schematic diagram of a device provided by an exemplary embodiment.

[0026] Fig. 9 A block diagram of a key-value pair storage device for graph data provided by an exemplary embodiment.

[0027] Fig.10 It is a block diagram of a graph data pre-fetching device provided by an exemplary embodiment. DETAILED DESCRIPTION

[0028] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with one or more embodiments of this specification. Instead, they are merely examples of devices and methods consistent with some aspects of one or more embodiments of this specification as detailed in the appended claims.

[0029] It should be noted that: in other embodiments, the steps of the corresponding method are not necessarily performed in the order shown and described in this specification. In some other embodiments, the steps included in the method may be more or less than those described in this specification. In addition, a single step described in this specification may be decomposed into multiple steps for description in other embodiments; and multiple steps described in this specification may be combined into a single step for description in other embodiments.

[0030] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this manual are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0031] Figure 1 FIG. 1 is a schematic diagram of an overall architecture provided by an exemplary embodiment. Figure 1 As shown, the architecture may include a graph database 11 and a data processing device 12. The graph database 11 may include one or more storage media, such as hard disk 111 and hard disk 112, and the data processing device 12 may include a cache 121 and a processing module 122. The processing module 122 may specifically be a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit). Of course, the data processing device also includes other necessary components, such as a bus, a network interface card, etc., which will not be repeated here.

[0032] When the processing module 122 in the data processing device 12 needs to process the graph data, it needs to first read the graph data related to the current processing operation from the hard disk (such as hard disk 111) equipped with the graph database 11. This data may be referred to as target data. In modern computer systems, in order to reduce the number of reads from the hard disk 111, when reading the target data, other data adjacent to the storage address of the target data will be read into the cache 121. When new target data needs to be read later, it will be searched from the data stored in the cache 121 first. If the corresponding data cannot be found, the data will be read from the hard disk 111.

[0033] In the process of analyzing based on graph data, after reading a certain category of target data, it is very likely to access other data of the same or related category. However, in the related art, the storage structure of the graph data is not organized, resulting in the data pre-fetched into the cache 121 not being the same or related to the target data category, which in turn leads to a low cache hit rate and the need for frequent hard disk read operations. As the scale of graph data continues to expand and the complexity of analysis requirements continues to grow, high-frequency hard disk read operations have significantly reduced overall performance.

[0034] Therefore, in order to solve the above problems, this specification proposes a key-value pair storage method for graph data, which can store the data of the composition elements in the graph in a centralized and orderly manner, and ensure that the key-value pairs of the composition elements of the same category can be continuously stored in adjacent physical locations in the storage medium. While reading the target data, if the data stored in the adjacent physical location is pre-read, the relevance of the pre-read data to the target data can be improved. Therefore, after caching these pre-read data, the hit rate of subsequent cached data can be improved, avoiding high-frequency hard disk reading operations, thereby greatly improving the reading efficiency of the graph data.

[0035] Figure 2 FIG. 1 is a flowchart of a key-value pair storage method for graph data provided by an exemplary embodiment. Figure 2 As shown, the method may include the following steps:

[0036] Step 201, obtain multiple composition element key-value pairs, in any composition element key-value pair: its value corresponds to the composition element data of the corresponding composition element in the graph data, its key contains multiple identification fields and one of the identification fields is the category identification of the corresponding composition element; wherein the composition elements corresponding to the multiple composition element key-value pairs include nodes and / or edges.

[0037] Specifically, the graph structure may include one or more graph elements, and any graph element may be any node in the graph or any edge in the graph. In specific applications, any node may be used to represent various entities, such as users, companies, places, etc., and each entity may also have its own information, such as attributes, levels, etc.; and edges may be used to represent the connection between different entities, that is, relationships, such as "User A (entity / node) lives in (relationship / edge) Beijing (entity / node)", "There is a transaction (relationship / edge) between account x1 (entity / node) and account x2 (entity / node)", etc. More specifically, each node can be distinguished according to the node type of the node contained in the graph, and each edge can be distinguished according to the edge type of the edge contained in the graph. Figure 3 FIG. 1 is a schematic diagram of a graph structure provided by an exemplary embodiment. Figure 3As shown, the graph may contain one or more nodes, corresponding to the entities represented by them, such as user A (node ​​311), Beijing (node ​​312), account x1 (node ​​313), account x2 (node ​​314), etc., and may also contain edges, corresponding to the relationships between the entities represented by them, such as "living in" (edge ​​321), "there is a transaction" (edge ​​322), etc. Among them, the entity represented by node 311 is user A, and the entity represented by node 312 is Beijing, so node 311 and node 312 are different types of nodes; the relationship represented by edge 321 is "living in", and the relationship represented by edge 322 is "there is a transaction", so edge 321 and edge 322 are different types of edges.

[0038] In one embodiment, any composition element in the graph and its corresponding data are represented and stored in the form of a composition element key-value pair. The composition element corresponding to any composition element key-value pair in the multiple composition element key-value pairs can be a node in the graph or an edge in the graph. All composition elements corresponding to the multiple composition element key-value pairs obtained can include nodes and edges in the graph at the same time.

[0039] Specifically, in any composition element key-value pair, the value of the composition element key-value pair is the composition element data of the corresponding composition element. When the composition element is any node, the value of the composition element key-value pair is the node data of the node; when the composition element is any edge, the value of the composition element key-value pair is the edge data of the edge. At the same time, for the value of any composition element key-value pair, the corresponding composition element data can also be organized through a variety of data structures, such as structures, arrays, etc., or one or more key-value pairs can be nested again to record the attributes, weights and other information of the composition element. Technicians can make choices based on actual conditions and needs, and this specification does not limit this.

[0040] The key of any composition element key-value pair contains multiple identification fields, and each identification field can represent the relevant information of the composition element in the graph, such as clarifying the position of the composition element in the graph and the relationship between the composition element and other composition elements, etc., to facilitate subsequent queries. In this embodiment, one of the fields in the key of any composition element key-value pair can be the category identifier of the corresponding composition element. The category identifier is used to represent the identifier used by technicians to classify composition elements in a preset dimension, which is different from the node type determined based on the type of entity represented by the node and the edge type determined based on the type of relationship represented by the edge. Specifically, the node type and edge type are used to represent the entity type represented by the corresponding node or edge, as mentioned above. Figure 3In the embodiment, node 311 represents user A and is a node of the "user" type, and node 313 represents Beijing and is a node of the "place" type. At this time, the composition element key-value pair of the node will include an identification field for representing the entity type, and will also include an identification field for representing other information (such as element type). The category identifier is not equivalent to the identification field described herein for representing the entity type. When the technician sets the preset dimension to the entity type represented by the composition element, the category identifier can be the identification field described herein for representing the entity type, and when the technician sets the preset dimension to the element type, the category identifier can be used to represent the field of the element type.

[0041] In one embodiment, the category identifier can be used as the first identification field of the key. However, it should be noted that when defining the first identification field, the sorting mechanism used by the storage engine when storing the key-value pair should be taken into account. When sorting multiple key-value pairs, if the storage engine takes the leftmost byte contained in the key as the starting byte and follows the rule of aggregating keys with the same starting byte, the first identification field should be the identification field located at the leftmost side, that is, the category identifier should be located at the leftmost side of the key; conversely, if the storage engine takes the rightmost byte contained in the key as the starting byte and follows the rule of aggregating keys with the same starting byte, the first identification field should be the identification field located at the rightmost side, that is, the category identifier should be located at the rightmost side of the key. For the convenience of description, the following embodiments of this specification will assume that the storage engine follows the first rule (that is, the position of the starting byte is the front, and the category identifier should be located at the leftmost side of the key) for description. Of course, the category identifier can also be used as the middle identification field of the key, that is, other identification fields can exist on the left and right sides of the category identifier field. In this case, the technician can determine the corresponding byte according to the specific position of the category identifier in the key, and reset the sorting mechanism used by the storage engine when storing the key-value pair according to the determined bytes, so that the storage engine can sort multiple keys according to the category identifier in the key. For example, if the category identifier is the second identification field in the key, and any identification field corresponds to four bytes, then the number of bytes corresponding to the category identifier is 5-8. At this time, the storage engine can be set to sort multiple keys according to the 5th to 8th bytes of the key.

[0042] It should also be noted that graph data is usually stored in a remote repository. In order to avoid the graph data analysis system frequently reading data from the remote repository during the analysis process, the analysis system will pre-acquire the data that may be needed during the analysis process and store it locally. Therefore, the multiple key-value pairs of composition elements acquired by the analysis system for subsequent storage may be organized by different entities.

[0043] In one embodiment, the graph data in the remote repository is not organized in the form of key-value pairs. For example, the remote repository can store graph data through a property graph model or an RDF (Resource Description Framework) model. In this case, after the graph data analysis system obtains the graph data from the remote, it needs to organize it into the form of key-value pairs and make it conform to the structure described in the above embodiment. In another embodiment, the remote repository may have organized the graph data into the form of key-value pairs when storing the graph data, and the key of the key-value pair conforms to the structure described in the above embodiment. In this case, after the analysis system obtains multiple composition element key-value pairs from the remote repository, it can directly store the multiple composition element key-value pairs in order according to the arrangement order between the category identifiers contained in each key.

[0044] Step 202 : storing the composition element key-value pairs in the plurality of composition element key-value pairs in order according to the arrangement order of the category identifiers contained in the corresponding keys.

[0045] In one embodiment, the storage engine can use its built-in sorting algorithm to optimize the arrangement logic of the key-value pairs according to the category identifiers contained in each key. Specifically, the storage engine can organize the keys according to specific rules (such as numerical order, dictionary order, etc.) through built-in data structures and technologies such as B+ trees and skip lists, so as to arrange the key-value pairs in a certain order. In the case where the category identifier is used as the first identification field of the key, the first field in the key can be used as the basis for sorting. For example, if the data type of the key is defined as a string type during storage, it can be sorted first by the first letter, and then by the subsequent letters; and if the data type of the key is converted into a byte array before storage, it can be sorted first by the value of the first byte, and then by the value of the subsequent bytes. On this basis, key-value pairs with the same category can be aggregated together and stored centrally, thereby ensuring that the key-value pairs of the same or similar categories of composition elements can be stored in adjacent physical locations in the storage medium. For storage engines that support batch operations, key-value pairs of composition elements of the same or similar categories may be further integrated and written to the storage medium in batches to further improve the writing efficiency during storage.

[0046] It can be seen from the above embodiments that the technical solution provided in this specification organizes the form of the key containing multiple identification fields in each composition element key-value pair, takes the category identification as one of the identification fields, and then stores the data of the composition elements in the figure in a centralized and orderly manner according to the arrangement order between the category identifications contained in the multiple keys, ensuring that the composition element key-value pairs of the same category can be continuously stored in adjacent physical locations in the storage medium. While reading the target data, if the data stored in the adjacent physical location is pre-read, the correlation between the pre-read data and the target data can be improved. Therefore, after caching these pre-read data, the hit rate of subsequent cached data can be improved, avoiding high-frequency hard disk reading operations, thereby greatly improving the reading efficiency of the image data.

[0047] In one embodiment, the identification fields included in the keys of different composition element key-value pairs are different, so the identification fields that can be used as category identifications in the keys of different composition element key-value pairs are also different.

[0048] First of all, it should be noted that when reading data, the graph data analysis system may query a specific node, but usually will not query a specific edge. When it is necessary to query edges, it is usually necessary to read all edges that meet the conditions by limiting the edge type, directionality, and nodes connected to the edge. Therefore, for any node, it is necessary to give it unique identification information, such as a node ID, but for any edge, it is not necessary to give it unique identification information.

[0049] Therefore, when the composition element is a node, the identification field may include a node type and a node identification. The node type is divided based on the type of entity represented by the corresponding node, and the node identification is used to represent the unique symbolic information of the node. In this case, the category identification may include the node type, and correspondingly, the other identification fields in the key may include the node identification. Figure 3 , Figure 3 The entity represented by node 311 is user A, so the node type of node 311 is "user". If the identifier corresponding to the node type "user" is "25", then the category identifier of node 311 is "25". The node identifier can be the ID of the node, such as "311". At this time, the node type "25" and the node identifier "311" are concatenated to obtain "25-311", and the key of the key-value pair of the composition element (node ​​311) can be obtained.

[0050] When the composition element is an edge, the identification field may include the edge type, directionality, and connected nodes, and any of the above identification fields can be used as a category identifier. Among them, the nodes connected by any edge are the two nodes directly connected by the edge, and the directionality of the edge is used to mark the relationship from one node to another. Of course, although there are undirected edges without a specific direction in graph theory, in practical applications, directed edges are usually used to represent the relationship between entities. In directed edges, the edge can be further clarified as an outgoing edge, an incoming edge, and a bidirectional edge based on the nodes it is connected to. It should be noted here that a single edge can have dual roles, being both an incoming edge of a node and an outgoing edge of another node. For example Figure 3 As shown, edge 321 is both an outgoing edge of node 311 and an incoming edge of node 312. In this case, the embodiment of this specification will store the edge in different directions. Then, for any node connected to the edge, all related edges can be found based on the node to form a more coherent edge data group, thereby improving subsequent reading efficiency. The edge type is divided based on the type of relationship represented by the corresponding edge. For example, the aforementioned Figure 3 The relationship represented by edge 321 is "living in", so the edge type of edge 321 is "living", and the relationship represented by edge 322 is "existing transaction", so the edge type of edge 322 is "transaction". Here, the edge type and directionality can be identified by corresponding identifiers, and the connected nodes can be identified by the unique symbolic information of the node, such as the node identifier.

[0051] Specifically, when the graph element is an edge, the technician can select the specific identifier referred to by the type identifier contained in the key in the key-value pair of the graph element according to the actual situation and needs. For example, if the analysis system of graph data usually follows the logic of "reading the node first, then reading the edge associated with the node" to read data, then the connected nodes can be used as the type identifier of the edge; if the analysis system usually follows the logic of "reading the node first, then reading a certain type of edge" to read data, then the edge type can be used as the type identifier of the edge. This specification does not limit this.

[0052] It should also be noted that, when the category identifier includes one of the above three (edge ​​type, directionality, and connected nodes), the other identification fields except the category identifier should be a combination of the remaining two, and do not include the information identifier corresponding to the category identifier. For example, if the connected nodes are used as the category identifier, then the other identification fields contained in the key should no longer include the connected nodes, and should be a combination of edge type and directionality; if the edge type is used as the category identifier, then the other identification fields contained in the key should no longer include the edge type, and should be a combination of directionality and connected nodes. At the same time, technicians can set the arrangement order of other identification fields contained in the key according to actual conditions and needs.

[0053] For example, Figure 3 The middle edge 321 is the outgoing edge of the node 311, and the relationship represented by the edge 321 is "residence", that is, the edge type of the edge 321 is "residence". Assuming that the identifier corresponding to the edge type "residence" is "90", the directionality identifier of the outgoing edge is "1", and the node identifier of the node 311 is "311", at this time, if the edge type is used as the category identifier, and the category identifier is used as the first identifier field of the key, and the directionality and the connected nodes are used as other identifier fields after the category identifier, and they are spliced, "90-1-311" or "90-311-1" can be obtained as the key of the key-value pair of the composition element (edge ​​321); and if the connected nodes are used as the category identifier, and the category identifier is used as the first identifier field of the key, and the edge type and directionality are used as other identifier fields after the category identifier, and they are spliced, "311-90-1" or "311-1-90" can be obtained as the key of the key-value pair of the composition element (edge ​​321). Similarly, the directionality of the edge can also be used as a category identifier, which will not be discussed here.

[0054] At the same time, it should be noted that in actual storage, it is necessary to convert each identifier contained in the key of the composition element key-value pair into a byte array, and the specific number of bytes can be set by the technician according to actual needs. For example, for the identifier "25" used to represent the node type, if the data type used in the declaration is a 4-byte int type, then after converting it into a byte array, it is [00000000, 00000000, 00000000, 00011001], and further in conjunction with the node identifier "311", "25-311" is converted into a byte array, and then [000000000, 00000000, 00000000, 00011001, 000000000, 00000000, 000000001, 00110111] can be obtained, which is the key actually used by the composition element key-value pair corresponding to node 311 when storing. Since byte array is the most basic data storage format in computers, this step is only used to illustrate the specific format used for storage and will not affect the aforementioned method of storing key-value pairs of composition elements.

[0055] In one embodiment, the key of the composition element key-value pair may also include the first element type identifier or the second element type identifier. Figure 4 As shown, when the composition element is a node, the key of the corresponding composition element key-value pair may also include a first element type identifier to indicate that the corresponding composition element is a node; when the composition element is an edge, the key of the corresponding composition element key-value pair may also include a second element type identifier to indicate that the corresponding composition element is an edge. It can be understood that when the graph includes both nodes and edges, it may be necessary to quickly distinguish between the composition element key-value pairs corresponding to the nodes and the composition element key-value pairs corresponding to the edges. Therefore, the corresponding element type identifier may be added to the key of the corresponding key-value pair to indicate that the composition element is a node or an edge. The first element type identifier and the second element type identifier may be used as category identifiers or as other identification fields. See. Figure 4 , if the category identifier is used as the first identifier field of the key, then in Figure 4 In the key-value pair of the composition element corresponding to the middle edge, the second element is identified as the category identifier in the key; at the same time, Figure 4 In the key-value pair of the composition element corresponding to the middle node, the node type is used as the category identifier, and the first element identifier is located after the category identifier, which is not limited in this specification.

[0056] It can be seen from the above embodiments that the technical solution provided in this specification organizes the form of the keys in each composition element key-value pair, and specifically can centrally and orderly store the composition element key-value pairs corresponding to nodes of the same node type, so that when the data is subsequently read, the node data of the same or similar type can be read into the cache together for use; at the same time, the technical solution provided in this specification, when storing the composition element key-value pairs corresponding to the edges, not only clarifies the type, directionality, and connected nodes of the edges, but also supports the selection of category division dimensions according to needs, and can use the identifiers of different information as category identifiers, so that it can be more adapted to the reading logic of the analysis engine and better meet the functional requirements of graph data.

[0057] In one embodiment, for any composition element, the corresponding data may change. For example, for a node of a certain user type, the node data includes level information, and the level information may change according to user behavior, such as upgrading or downgrading. When performing analysis, the graph analysis system does not necessarily analyze based on the latest data, but may also select data within a certain time period for analysis based on specific needs. Therefore, when the data of a composition element changes, the previous data should not be directly overwritten with the current data, but the new data should be stored in a new physical location to ensure that the old data can also be queried and used.

[0058] In this embodiment, a preset fixed time length can be used as a time window, and a time window identifier can be added to the key of the composition element key-value pair, and the value of the time window identifier can identify different time windows, such as Figure 5 As shown. In the case where the key of any composition element key-value pair contains a time window identifier, the value of the composition element key-value pair is the incremental data of the composition element data in the current time window compared to the composition element data in the previous time window. Incremental data is different from full data. It refers to the data part that is newly added on the basis of existing data, reflecting the latest changes or updates.

[0059] You can continue to see here Figure 3 . Figure 3The information stored for user A (node ​​311) in the example includes level information "3" and attribute information "adult". If the identifier of the current time window is "w1", and within the current time window, the user's level changes to "4" and is still "adult", then the incremental data corresponding to the current time window is level information "4". In this case, the incremental data (level information "4") is used as the value, and the key containing the time window identifier "w1" is used as the new key to store it again. Since the attribute information "adult" has not changed, it is not necessary to store the attribute information "adult" this time. At this time, for user A (node ​​311), two key-value pairs of composition elements are obtained: one is a key-value pair that does not contain the time window identifier in the key, and the information included in its value is level information "3" and attribute information "adult"; the other is a key-value pair that contains the time window identifier "w1" in the key, and the information included in its value is level information "4". Compared with storing the full amount of data, storing incremental data can effectively reduce the time of data processing and the occupation of storage space.

[0060] In one embodiment, since other data besides the graph data may be stored in the storage medium. It is understandable that, although the method of the aforementioned embodiment can store the key-value pairs of the composition elements centrally according to the category identifier, the key-value pairs of the composition elements corresponding to the nodes and the key-value pairs of the composition elements corresponding to the edges may still be stored in the storage medium in a dispersed manner. Therefore, in order to centrally store the data of all the composition elements contained in the current graph, the key-value pairs of the composition elements may also include placeholders. The placeholder is used to add to the key of the composition element key-value pair corresponding to the node and / or the key of the composition element key-value pair corresponding to the edge, so that the byte lengths of the two are adjusted to be consistent. In the embodiment of the specification, the placeholder may be located after the category identifier contained in the corresponding key to ensure that the storage engine still sorts and aggregates the key-value pairs of the composition elements based on the category identifier, and the organization order between the placeholder and other identification fields in the key can be set according to actual conditions and needs. Generally speaking, the placeholder is located after other identification fields.

[0061] For example, Figure 3 The key of the key-value pair of the composition element corresponding to the middle node 311 is "25-311", which specifically includes two identification fields, while the key of the key-value pair of the composition element corresponding to the edge 321 is "90-311-1", which specifically includes three identification fields. In this case, you can add a placeholder, such as "0", after the key of the key-value pair of the composition element corresponding to the node 311 to get "25-311-0", which is equivalent to the key also including three identification fields. At this time, Figure 6As shown, the length of the key-value pairs of the graph elements corresponding to the nodes is consistent with that of the key-value pairs of the graph elements corresponding to the edges. On this basis, when the storage engine sorts and stores each key-value pair of the graph elements, it can store the key-value pairs of the graph elements corresponding to the nodes and the key-value pairs of the graph elements corresponding to the edges in adjacent physical locations, thus realizing the centralized storage of graph data as a whole. Of course, in actual storage, the placeholders also need to be converted into byte arrays, which will not be described here.

[0062] Correspondingly, this specification also provides a graph data pre-fetching method. Figure 7 FIG. 1 is a flowchart of a graph data pre-fetching method provided by an exemplary embodiment. Figure 7 As shown, the method may include the following steps:

[0063] Step 701, receiving a graph data reading request for a graph database; the graph database stores a plurality of composition element key-value pairs, each composition element key-value pair is stored in order according to the arrangement order between the category identifiers contained in the corresponding key, and for any composition element key-value pair: its value corresponds to the composition element data of the corresponding composition element in the graph data, its key contains a plurality of identification fields and one of the identification fields is the category identifier of the corresponding composition element; wherein the composition elements corresponding to the plurality of composition element key-value pairs include nodes and / or edges.

[0064] Step 702: Determine a matching key and a storage address of a corresponding composition element key-value pair from the plurality of composition element key-value pairs.

[0065] Step 703 : for a pre-fetch address range that matches the determined storage address of the composition element key-value pair, read the composition element key-value pairs stored in the pre-fetch address range.

[0066] In the graph data pre-fetching method, the contents related to the composition element key-value pairs and the structure of the keys of the composition element key-value pairs can be found in the description of the aforementioned embodiments and will not be repeated here.

[0067] In one embodiment, the storage engine may determine the target composition element that the graph data analysis system wants to read based on a graph data read request to a graph database, and further determine the key of the key-value pair corresponding to the target composition element from multiple composition element key-value pairs, as well as the storage address of the composition element key-value pair. Here, the composition element that the analysis system wants to read may be a single composition element, such as "node 311", or may be multiple composition elements, such as "all outgoing edges of node 311". Accordingly, the number of keys that match the graph data read request and the corresponding storage addresses of the composition element key-value pairs determined from multiple composition element key-value pairs may also be one or more, and this application does not impose any restrictions on this.

[0068] In one embodiment, a pre-fetch address range that matches the determined storage address of the composition element key-value pair can be determined, and the composition element key-value pairs stored in the pre-fetch address range can be read. Here, the pre-fetch address range is usually a physical address range adjacent to the determined storage address, which is specifically related to the way modern computer systems read data. When a modern computer system reads data from a continuous address, it not only reads the data corresponding to the address, but also reads all the data of the data block where the address is located into the cache. Therefore, the size of the pre-fetch address range can be determined according to the size of each data block during actual storage. Of course, the pre-fetch address range can also be determined by the technician according to actual conditions and needs. For example, when the scale of the graph data is large and more data information is required for analysis, the pre-fetch address range can be appropriately expanded; when the scale of the graph data is small and less data information is required for analysis, the pre-fetch address range can be appropriately reduced, and this specification does not limit this.

[0069] In one embodiment, when the key of the composition element key-value pair also includes a time window identifier, the corresponding target time window identifier can be determined according to the target time range carried in the graph data reading request. For example, in the case of dividing the time window by days, if you want to obtain the data of the last three days of a node, assuming that the latest window identifier corresponding to the current time is "5", then the target time window identifier can be determined as "3", "4", "5". After that, the key corresponding to the composition element and containing the target time window identifier can be found from multiple composition element key-value pairs, and then the storage address of the composition element key-value pair corresponding to the key can be determined. Thereafter, the pre-fetch address range can be further determined according to the determined storage address, and the composition element key-value pairs stored in the pre-fetch address range can be read. It should be noted that for any composition element, after reading its data in multiple different time windows, it is necessary to integrate the multiple data and use the integrated value as the final usage data. The integration rules here can be set by the technician according to the needs, such as using the latest value as the integration value, the average value as the integration value, etc., and this application does not limit this.

[0070] In another embodiment, there is a situation where the key of the composition element key-value pair contains a time window identifier, but the image data reading request does not carry the target time range. In this case, the key-value pairs corresponding to all keys of the target composition element with the time window identifier can be read, and the data of each value can be integrated. The integration rules here can also be set by the technician according to the needs, and this application does not limit this.

[0071] The technical solution provided in this specification organizes the form of the key in each composition element key-value pair, and stores the data of the composition elements in the graph in a centralized and orderly manner, ensuring that the composition element key-value pairs of the same category can be continuously stored in adjacent physical locations in the storage medium, and then when reading data, the composition element key-value pairs within a certain adjacent physical location range can be read into the cache together. It can be understood that in the process of analysis based on graph data, after reading a certain category of composition element data, it is very likely to access other composition elements of the same or related category. The method provided in this specification can read related data of the same category into the cache together for subsequent use, thereby improving the cache hit rate, avoiding high-frequency hard disk read operations, and greatly improving the reading efficiency of graph data.

[0072] Figure 8 is a schematic structural diagram of a device provided by an exemplary embodiment. Figure 8 At the hardware level, the device includes a processor 802, an internal bus 804, a network interface 806, a memory 808, and a non-volatile memory 810, and may also include hardware required for other functions. One or more embodiments of this specification may be implemented based on software, such as the processor 802 reading the corresponding computer program from the non-volatile memory 810 into the memory 808 and then running it. Of course, in addition to the software implementation, one or more embodiments of this specification do not exclude other implementations, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0073] Please refer to Fig. 9 , the key-value pair storage device of graph data can be applied to Figure 8 The device shown in the figure is used to implement the technical solution of this specification. The key-value pair storage device of the graph data may include:

[0074] The acquisition unit 91 is used to acquire a plurality of composition element key-value pairs, wherein in any composition element key-value pair: its value corresponds to the composition element data of the corresponding composition element in the graph data, its key comprises a plurality of identification fields and one of the identification fields is a category identification of the corresponding composition element; wherein the composition elements corresponding to the plurality of composition element key-value pairs include nodes and / or edges;

[0075] The storage unit 92 is used to sequentially store each composition element key-value pair in the plurality of composition element key-value pairs according to the arrangement order between the category identifiers contained in the corresponding keys.

[0076] Optional,

[0077] In the case where the composition element is a node, the category identifier includes: a node type or a first element type identifier used to indicate that the corresponding composition element is a node;

[0078] In the case where the composition element is an edge, the category identifier includes: edge type, directionality, connected nodes, or a second element type identifier used to indicate that the corresponding composition element is an edge.

[0079] Optional,

[0080] In the case where the composition element is a node, the key of the corresponding composition element key-value pair further includes a first element type identifier, so as to indicate that the corresponding composition element is a node;

[0081] And / or, in the case where the composition element is an edge, the key of the corresponding composition element key-value pair further includes a second element type identifier, so as to indicate that the corresponding composition element is an edge.

[0082] Optionally, the key of at least one composition element key-value pair among the multiple composition element key-value pairs further includes a time window identifier;

[0083] In the case where the key of any composition element key-value pair includes the time window identifier, the value of any composition element key-value pair is: the incremental data of the composition element data of the corresponding composition element in the current time window compared to the composition element data in the previous time window, and the current time window is indicated by the value of the time window identifier.

[0084] Optionally, the composition element key-value pair also includes a placeholder; the placeholder is located after the category identifier contained in the corresponding key, and is used to be added to the key of the composition element key-value pair corresponding to the node and / or the key of the composition element key-value pair corresponding to the edge so that the byte lengths of the two are adjusted to be consistent.

[0085] Please refer to Fig.10 , the graph data pre-fetching device can be applied to Figure 8 The device shown in the figure is used to implement the technical solution of this specification. The image data pre-fetching device may include:

[0086] A receiving unit 101 is used to receive a graph data reading request for a graph database; the graph database stores a plurality of composition element key-value pairs, each composition element key-value pair is stored in order according to the arrangement order between the category identifiers contained in the corresponding key, and for any composition element key-value pair: its value corresponds to the composition element data of the corresponding composition element in the graph data, its key contains a plurality of identification fields and one of the identification fields is the category identifier of the corresponding composition element; wherein the composition elements corresponding to the plurality of composition element key-value pairs include nodes and / or edges;

[0087] A determination unit 102, configured to determine a matching key and a storage address of a corresponding composition element key-value pair from the plurality of composition element key-value pairs;

[0088] The reading unit 103 is used to read the composition element key-value pairs stored in the pre-fetch address range that matches the determined storage address of the composition element key-value pairs.

[0089] Optional,

[0090] In the case where the composition element is a node, the category identifier includes: a node type, a node identifier, or a first element type identifier used to indicate that the corresponding composition element is a node;

[0091] In the case where the composition element is an edge, the category identifier includes: edge type, directionality, connected nodes, or a second element type identifier used to indicate that the corresponding composition element is an edge.

[0092] Optionally, the key of the composition element key-value pair further includes a time window identifier;

[0093] The step of determining a matching key and a storage address of a corresponding composition element key-value pair from the plurality of composition element key-value pairs includes:

[0094] Determine a corresponding target time window identifier according to the target time range carried in the graph data reading request;

[0095] A key including the target time window identifier is searched from the multiple composition element key-value pairs, and then a storage address of the composition element key-value pair corresponding to the key is determined.

[0096] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative work.

[0097] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, which may be in the form of a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email transceiver, a game console, a tablet computer, a wearable device or a combination of any of these devices.

[0098] Based on the same concept as the above method, this specification also provides an electronic device, including: a processor; a memory for storing processor executable instructions; wherein the processor implements the steps of the method described in any of the above embodiments by running the executable instructions.

[0099] Based on the same concept as the above method, this specification also provides a computer-readable storage medium on which computer instructions are stored. When the instructions are executed by a processor, the steps of the method described in any of the above embodiments are implemented.

[0100] Based on the same concept as the above method, this specification also provides a computer program product, including a computer program / instruction, which implements the steps of the method described in any of the above embodiments when executed by a processor.

Claims

1. A key-value pair storage method for graph data, characterized in that: include: Acquire multiple composition element key-value pairs, in which: in any composition element key-value pair: its value corresponds to the composition element data of the corresponding composition element in the graph data, its key includes multiple identification fields and one of the identification fields is the category identification of the corresponding composition element; wherein the composition elements corresponding to the multiple composition element key-value pairs include nodes and / or edges; The composition element key-value pairs in the plurality of composition element key-value pairs are stored in order according to the arrangement order between the category identifiers contained in the corresponding keys.

2. The method according to claim 1, characterized in that In the case where the composition element is a node, the category identifier includes: a node type or a first element type identifier used to indicate that the corresponding composition element is a node; In the case where the composition element is an edge, the category identifier includes: edge type, directionality, connected nodes, or a second element type identifier used to indicate that the corresponding composition element is an edge.

3. The method according to claim 1, characterized in that The key of at least one composition element key-value pair among the plurality of composition element key-value pairs further comprises a time window identifier; In the case where the key of any composition element key-value pair includes the time window identifier, the value of any composition element key-value pair is: the incremental data of the composition element data of the corresponding composition element in the current time window compared to the composition element data in the previous time window, and the current time window is indicated by the value of the time window identifier.

4. The method according to claim 1, characterized in that: The composition element key-value pair also includes a placeholder; the placeholder is located after the category identifier contained in the corresponding key, and is used to be added to the key of the composition element key-value pair corresponding to the node and / or the key of the composition element key-value pair corresponding to the edge so that the byte lengths of the two are adjusted to be consistent.

5. A graph data pre-fetching method, characterized in that: include: Receive a graph data reading request for a graph database; The graph database stores a plurality of composition element key-value pairs, each composition element key-value pair is stored in order according to the arrangement order between the category identifiers contained in the corresponding key, and for any composition element key-value pair: its value corresponds to the composition element data of the corresponding composition element in the graph data, its key contains a plurality of identification fields and one of the identification fields is the category identifier of the corresponding composition element; wherein the composition elements corresponding to the plurality of composition element key-value pairs include nodes and / or edges; Determine a matching key and a storage address of a corresponding composition element key-value pair from the plurality of composition element key-value pairs; For a pre-fetch address range that matches the determined storage address of the composition element key-value pair, the composition element key-value pairs stored in the pre-fetch address range are read.

6. The method according to claim 5, characterized in that In the case where the composition element is a node, the category identifier includes: a node type, a node identifier, or a first element type identifier used to indicate that the corresponding composition element is a node; In the case where the composition element is an edge, the category identifier includes: edge type, directionality, connected nodes, or a second element type identifier used to indicate that the corresponding composition element is an edge.

7. The method according to claim 5, characterized in that The key of the composition element key-value pair also includes a time window identifier; The step of determining the key of the matching composition element key-value pair and the corresponding composition element key-value pair storage address from the multiple composition element key-value pairs includes: Determine a corresponding target time window identifier according to the target time range carried in the graph data reading request; A key including the target time window identifier is searched from the multiple composition element key-value pairs, and then a storage address of the composition element key-value pair corresponding to the key is determined.

8. An electronic device, characterized in that: include: processor; A memory for storing processor-executable instructions; wherein the processor implements the steps of the method according to any one of claims 1 to 7 by executing the executable instructions.

9. A computer-readable storage medium, characterized in that: Computer instructions are stored thereon, and when the instructions are executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer program product, characterized in that The method comprises a computer program / instruction, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 7.

Images