Graph index calculation method and device for graph data

Abstract

Embodiments of the present specification disclose a graph index calculation method and device for graph data, wherein the graph data is divided into K shards and stored in a graph data warehouse. The method is executed by a first device and includes: in response to triggering of a dump condition, executing a first data dump task for data dumping on a first shard of the K shards; wherein executing the first data dump task includes: in the process of reading first shard data corresponding to the first shard from the graph data warehouse, performing streaming statistical calculation related to nodes and / or edges on the read data segments to obtain a first sub-index value of a target graph index corresponding to the first shard; and the first sub-index value is used for aggregating sub-index values corresponding to other shards to obtain a target index value.

Description

Technical Field

[0001] This specification relates to the field of computer technology, and in particular to a method and apparatus for calculating the spectral index of graph data, a computer-readable storage medium, and a computing device. Background Technology

[0002] A relational network graph (also known as a relational map, semantic graph, or knowledge graph) is a description of the relationships between entities in the real world and is widely used in various computer information processing applications. Generally, a relational network graph contains a set of nodes and a set of edges. Nodes represent entities in the real world, and edges represent the connections between entities. For example, in a social network, people are entities, and the relationships or connections between people are edges.

[0003] In many business scenarios, it is necessary to observe graph metrics. For example, for social networks built on social data collected after user authorization on social platforms, it is necessary to calculate knowledge popularity metrics to observe popular users in different fields, in order to recommend to potential fans or conduct advertising in related fields.

[0004] However, with the advent of the big data era, graph data has also experienced explosive growth and updates. Therefore, there is an urgent need for a solution that can effectively reduce the computational load of graph indicators and improve the timeliness of graph indicator calculation. Summary of the Invention

[0005] This specification describes a method and apparatus for calculating spectral indices of graph data, which can effectively reduce the computational load of spectral indices and improve the timeliness of spectral indices calculation.

[0006] According to a first aspect, a method for calculating graph metrics of graph data is provided, wherein the graph data is divided into K fragments and stored in a graph data warehouse; the method is executed by a first device and includes:

[0007] In response to the triggering of the dump condition, a first data dump task is executed. The first data dump task is used to dump data for the first shard among the K shards. Executing the first data dump task includes: during the process of reading the first shard data corresponding to the first shard from the graph data warehouse, performing streaming statistical calculations related to nodes and / or edges on the read data fragments to obtain a first sub-index value of the target graph index corresponding to the first shard; the first sub-index value is used to aggregate with the sub-index values ​​corresponding to other shards to obtain the target index value.

[0008] In one embodiment, performing a first data dump task includes executing the first data dump task in parallel with other K-1 data dump tasks.

[0009] In one embodiment, the K segments are obtained based on a point-splitting mechanism. The first segment includes multiple one-hop subgraphs, wherein the data of each one-hop subgraph includes the point data of the central node and the edge data of the connecting edges connected to the central node.

[0010] In one embodiment, the target graph index includes knowledge popularity, which indicates the N nodes with the most edges among the target type nodes; wherein, performing streaming statistical calculations related to nodes and / or edges on the read data segments to obtain the first sub-index value of the target graph index corresponding to the first segment includes:

[0011] Based on the sequentially read data fragments related to the first node, a first degree of the first node is calculated, wherein the first node belongs to the target type. The first degree is inserted into a priority queue data structure corresponding to the target type, so that the degree inserted into the priority queue data structure dynamically eliminates node degrees outside the top N positions. In response to the completion of reading the first data fragment, the N node degrees stored in the priority queue data structure are used as the first sub-index value of the knowledge popularity.

[0012] In a specific embodiment, the first degree of the first node is calculated based on several data segments related to the first node read sequentially, including:

[0013] In response to reading a first data segment, a counter value is set for a first node; wherein the first data segment includes first attribute data of the first node, and a first number of edge information following the first attribute data as the segment end; the counter value is the first number. In response to reading at least one second data segment, wherein each second data segment includes a predetermined indicator and a second number of edge information, the respective second numbers are added to the counter value; wherein the predetermined indicator indicates that the node information has not changed. In response to reading attribute data of a second node in a data segment, the current counter value is used as the first degree.

[0014] In another specific embodiment, based on several data segments related to the first node read sequentially, the first degree of the first node is calculated, including:

[0015] In response to reading a first data segment, wherein the data entries include at least the attribute data of a first node, the edge information of a first number, and the attribute data of a second node in sequence, the first number is determined as a first degree.

[0016] On the other hand, in a specific embodiment, the method further includes: based on the first sub-index value of the knowledge popularity and the other sub-index values ​​of the knowledge popularity determined in the other K-1 data dumping tasks, the degree of the top N nodes in K sets is accumulated and sorted by node to obtain the N nodes corresponding to the final degree of the top N nodes, which are used as the target index value of the knowledge popularity.

[0017] In one embodiment, the target graph metric includes the outgoing / incoming edge mounting rate; wherein, performing streaming statistical calculations related to nodes and / or edges on the read data segments to obtain a first sub-metric value of the target graph metric corresponding to the first segment includes:

[0018] Using data fragments, update the deduplication start / endpoint counts of the target edge type, and update the node counts of the corresponding start / endpoint types of the target edge type. In response to the completion of reading the first fragment data, obtain a first sub-index value of the outgoing edge mounting rate based on the deduplication startpoint count of the target edge type and the node count of the corresponding startpoint type, and / or obtain a first sub-index value of the incoming edge mounting rate based on the deduplication endpoint count of the target edge type and the node count of the corresponding endpoint type.

[0019] In one specific embodiment, updating the deduplication start / end point count of the target edge type using data fragments includes: inserting the start / end point identifiers of the connecting edges with the target edge type in the data fragments into a probability data structure of HyperLogLog or HyperLogLog plusplus to obtain the updated deduplication start / end point count corresponding to the target edge type.

[0020] In another specific embodiment, the method further includes:

[0021] Based on the first sub-index value of the outbound / inbound edge loading rate, and the other sub-index values ​​of the outbound / inbound edge loading rate determined in the other K-1 data dumping tasks, the ratio of the deduplication start point count and the corresponding start point node count of the target edge type is calculated as the target index value of the outbound edge loading rate, and / or the ratio of the deduplication end point count and the corresponding end point node count of the target edge type is calculated as the target index value of the inbound edge loading rate.

[0022] In one embodiment, the target graph index includes an invalid edge index, which indicates the number of invalid edges; wherein, performing streaming statistical calculations related to nodes and / or edges on the read data segments to obtain a first sub-index value of the target graph index corresponding to the first segment includes:

[0023] Based on whether the point attribute data of the start or end point of each connection edge in the data segment is empty, update the count of invalid edges; in response to the completion of reading the first fragment data, determine the first sub-index value of the invalid edge index based on the updated invalid edge count.

[0024] In one specific embodiment, the K shards are obtained based on a point splitting mechanism; the method further includes: summing the first sub-index value of the invalid edge index and the other sub-index values ​​of the invalid edge index determined in the other K-1 data dumping tasks to obtain the target index value of the invalid edge index.

[0025] In one embodiment, the target graph index includes a degree distribution index; wherein, based on the read data segments, streaming computation related to nodes and / or edges is performed to obtain a first sub-index value of the target graph index corresponding to the first segment, including:

[0026] Based on the degree of each node with a target point type determined from the data fragments, the degree distribution corresponding to the target point type is updated. In response to the completion of reading the first fragment of data, the updated degree distribution corresponding to the target point type is assigned to the first sub-index value of the degree distribution index.

[0027] In a specific embodiment, updating the degree distribution corresponding to the target point type based on the degree of each node with the target point type determined based on the data fragment includes:

[0028] Parse the data segment to find several first nodes of the target point type, and find several first degrees corresponding to these first nodes that are within a preset degree threshold window; insert these first degrees into the CountMinSketch data structure to obtain the updated degree distribution corresponding to the target point type. And / or, parse the data segment to find several second nodes of the target point type, and find several second degrees corresponding to these second nodes that are outside the degree threshold window; update the multidimensional degree-count tuple using these second degrees.

[0029] In a more specific embodiment, the method further includes: processing the plurality of first degrees using a Bitmap data structure to update the bitmap corresponding to the target point type for marking different degrees.

[0030] On the other hand, in a specific embodiment, the method further includes: aggregating the first sub-index value of the degree distribution index and other sub-index values ​​of the degree distribution index determined in the other K-1 data dumping tasks according to different degrees under the target point type to obtain the target index value of the degree distribution index.

[0031] In one embodiment, the target graph features include attribute coverage; wherein, based on the read data fragments, streaming computation related to nodes and / or edges is performed to obtain a first sub-index value of the target graph index corresponding to the first segment, including:

[0032] Based on the data fragment, update the attribute distribution and total number of nodes corresponding to the target point type, wherein the attribute distribution includes the number of nodes whose attribute values ​​are not empty for each attribute under the target point type; in response to the completion of reading the first data fragment, assign the updated attribute distribution and total number of nodes to the first sub-index value of the attribute coverage.

[0033] In one specific embodiment, the method further includes: aggregating the attribute coverage based on the first sub-index value of the attribute coverage and other sub-index values ​​of the attribute coverage determined in the other K-1 data dumping tasks to obtain the aggregated attribute distribution and the total number of aggregated nodes corresponding to the target point type; for each attribute in the aggregated attribute distribution corresponding to the target point type, dividing the number of nodes with non-empty attribute values ​​under that attribute by the total number of aggregated nodes corresponding to the target point type to obtain the target attribute coverage of the target attribute.

[0034] According to a second aspect, a graph index calculation device is provided for graph data, wherein the graph data is divided into K fragments and stored in a graph data warehouse; the device is integrated into a first device and includes: a dump module configured to execute a first data dump task in response to the triggering of a dump condition, the first data dump task being used to dump data for the first fragment among the K fragments; the index calculation unit included in the dump module is configured to: in the process of reading the first fragment data corresponding to the first fragment from the graph data warehouse, perform streaming statistical calculations related to nodes and / or edges on the read data fragments to obtain a first sub-index value of the target graph index corresponding to the first fragment; the first sub-index value is used to aggregate with the sub-index values ​​corresponding to other fragments to obtain the target index value.

[0035] According to a third aspect, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the method provided in the first aspect.

[0036] According to a fourth aspect, a computing device is provided, including a memory and a processor, wherein the memory stores executable code, and the processor, when executing the executable code, implements the method provided in the first aspect.

[0037] In summary, by employing the methods or apparatus disclosed in the embodiments of this specification, and by reusing the dump link, the sub-index values ​​of the map index are calculated in a streaming manner within a dump task targeting a single map data fragment, and then the values ​​of each sub-index are merged to obtain the complete index value. This effectively reduces the computational load and computation time of the map index, significantly improving computational efficiency. Attached Figure Description

[0038] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0039] Figure 1 This is a diagram illustrating an implementation scenario of the graph index calculation scheme for graph data disclosed in the embodiments of this specification.

[0040] Figure 2 This illustrates the point segmentation of the graph data;

[0041] Figure 3 This illustrates edge partitioning of the graph data;

[0042] Figure 4 This is a schematic diagram of the flowchart steps for calculating the spectral index of the graph data disclosed in the embodiments of this specification;

[0043] Figure 5 A schematic diagram of the process steps for calculating the first sub-index value of knowledge popularity as disclosed in the embodiments of this specification;

[0044] Figure 6 This is a schematic diagram of the storage structure of a one-hop subgraph data disclosed in the embodiments of this specification;

[0045] Figure 7 This is a schematic diagram of the process steps for calculating the first sub-index value of the outbound / inbound edge mounting rate as disclosed in the embodiments of this specification.

[0046] Figure 8 This is a schematic diagram of the process steps for calculating the first sub-index value of the invalid edge index as disclosed in the embodiments of this specification.

[0047] Figure 9 This is a schematic diagram of the process steps for calculating the first sub-index value of the degree distribution index as disclosed in the embodiments of this specification.

[0048] Figure 10 This is a schematic diagram of the process steps for calculating the first sub-index value of attribute coverage as disclosed in the embodiments of this specification.

[0049] Figure 11This is a schematic diagram of the structure of the graph index calculation device for the graph data disclosed in the embodiments of this specification. Detailed Implementation

[0050] The solution provided in this specification will now be described with reference to the accompanying drawings.

[0051] As mentioned earlier, many business scenarios require the observation of graph metrics. One method of calculating graph metrics relies on a full offline data table of the graph data. This involves repeatedly performing operations such as joining point-edge tables, table traversal, and executing user-defined functions (UDFs) when calculating various metrics, resulting in significant computational redundancy. Another method can calculate metrics based on graph computation; however, it cannot solve the problem of redundant graph construction, requiring the full graph data to be imported into the graph computation system before graph features can be calculated.

[0052] Furthermore, the applicant discovered that graph data repositories, such as data warehouses or databases storing graph data, perform graph construction and dump tasks periodically (e.g., daily) because the source data used to construct the graph is constantly updated. For example, new social data is continuously generated on social platforms: user U1 follows user U2, user U1 unfollows user U3, user U4's account is deleted, etc. Graph construction includes reconstructing the graph using the newly generated data, thereby dumping the full graph data of the newly constructed graph.

[0053] When the total volume of map data is large (e.g., hundreds of millions or trillions), the map data warehouse will slice the total map data before storing it. Accordingly, in the dumping task, it is necessary to first read each map data slice and then store it on other devices or other areas of the same device.

[0054] Based on the above observations and analysis, the applicant proposes a map index calculation scheme. By reusing the dump link, the sub-index values ​​of the map index are calculated in a streaming manner during the dump task for a single map data fragment. Then, the values ​​of each sub-index are merged or aggregated to obtain the complete index value. In this way, the computational workload of the map index can be effectively reduced, the computation time can be shortened, and the computation efficiency can be greatly improved.

[0055] Figure 1 This diagram illustrates an implementation scenario of the graph index calculation scheme for graph data disclosed in the embodiments of this specification, showing the division of the complete graph data into multiple ( Figure 1The diagram shows K shards that are dumped to K distributed nodes. During the dumping of each shard, streaming computation is performed on the different data segments read sequentially to obtain the sub-index values ​​of the graph index. After the data is read, all K sub-index values ​​of the graph index are merged to obtain the complete index value, or the final index value or target index value.

[0056] It should be noted that, Figure 1 The illustration shows the dumping of graph data to K different work nodes. In fact, the embodiments in this specification do not limit the target device or target location for dumping. For example, it can also be that K fragments are dumped to one or other numbers of work nodes.

[0057] Furthermore, for ease of understanding, before introducing the specific implementation steps of the above-mentioned graph index calculation scheme, we will first introduce the method for generating K fragments in the graph data warehouse, that is, the mechanism for splitting graph data. First, it is important to understand that the graph data splitting operation can be performed by the graph data warehouse, or by other devices or servers with computing and processing capabilities.

[0058] Mechanisms for segmenting graph data include point segmentation and edge segmentation. In the embodiments disclosed in this specification, the graph indicators to be observed can be selected and configured in advance as needed.

[0059] Figure 2 This illustrates the point partitioning of graph data. The entire graph is divided into three subgraphs based on node A. Node A is then stored in all three subgraphs, and the connecting edges are distributed and stored independently in all three subgraphs. This needs to be understood. Figure 2 The graph illustrated is quite simple; in real-world business scenarios, the graphs are far more complex, often containing hundreds of millions or even trillions of nodes and edges. For example, a DistrubitedNE sharding scheme can be used to achieve node partitioning.

[0060] Figure 3 This diagram illustrates edge partitioning of graph data, dividing the entire graph into two subgraphs along edges AB, AC, AD, and AE. To preserve the integrity of both subgraphs, edges AB, AC, AD, and AE need to be stored simultaneously in both subgraphs. For example, the METIS partitioning scheme can be used to implement edge partitioning.

[0061] The above describes the partitioning mechanism upon which the K partitions are generated. Next, we will combine... Figure 4 The specific implementation steps of the above-described spectral index calculation scheme are described in further embodiments.

[0062] Figure 4This is a schematic diagram of the process steps for calculating the spectral index of the graph data disclosed in the embodiments of this specification, wherein the graph data is divided into K segments and stored in the spectral data warehouse. Figure 4 The illustrated method is executed through the first device. It should be noted that the "first" in "first device" and similar terms such as "first" and "second" elsewhere in the text are used to distinguish similar things and do not have any other limiting function such as ordering. In addition, the first device can be a working node in a distributed computing system, or it can be a storage device other than the graph data warehouse, or it can be a backup system of the graph data warehouse.

[0063] like Figure 4 As shown, the method includes the following steps:

[0064] First, in step S410, in response to the triggering of the dump condition, a first data dump task is executed, which is used to dump data for the first of the K shards.

[0065] The above-mentioned dump conditions can be configured in advance as needed. In one embodiment, the dump condition includes a predetermined duration for a cyclic timer. In another embodiment, the dump condition includes the current time being a predetermined time of day. In yet another embodiment, the dump condition includes the completion of graph data segmentation for the newly constructed map. In still another embodiment, the dump condition includes receiving a dump operation instruction, for example, receiving a click instruction from a staff member on the dump icon in the system interface.

[0066] In response to the aforementioned dump conditions, K dump tasks, including the first data dump task, can be executed. In one embodiment, with the support of hardware resources, the first data dump task can be executed in parallel with the other K-1 data dump tasks. In another embodiment, the first data dump task can also be executed serially or in batches with the other K-1 data dump tasks.

[0067] It should be understood that the first data dump task can refer to any one of the K dump tasks, and there is a one-to-one correspondence between the K dump tasks and the K shards. In this paper, the first shard refers to one of the K shards used for dumping by the first data dump task. For example, the first data dump task is used to dump the first shard from the graph data warehouse to the first working node among the K working nodes.

[0068] The above outlines the basics of the first data dump task. Further, the following additional steps are designed to be performed within the first data dump task.

[0069] Specifically, during the process of reading the first piece of data corresponding to the first piece from the graph database, streaming statistical calculations related to nodes and / or edges are performed on the read data fragment (or current data fragment) to obtain the first sub-index value of the target graph index corresponding to the first piece.

[0070] It should be noted that, under normal circumstances, even if the complete map data is fragmented, the amount of data in a single fragment is still relatively large, while the amount of data read and transmitted each time is limited. Therefore, data is read and transmitted in batches, and the data read in a batch may be referred to as a data fragment in this article.

[0071] The graph data required for calculating different graph metrics may vary. For example, some only require vertex data, some only require edge data, and some require both vertex and edge data. Accordingly, streaming statistical calculations related to nodes, edges, or nodes and edges can be performed on the read data segments.

[0072] It is understandable that the above-mentioned streaming statistical computation is a composite expression of streaming computation and statistical computation. In contrast to streaming computation is batch computation, which requires data accumulation before batch processing, while streaming computation can process data as it arrives, effectively reducing processing latency. Statistical computation involves the effective collection, organization, and analysis of data with randomness.

[0073] By performing streaming statistical calculations on the read data segments, the first sub-index value corresponding to the first segment of the target map index can be obtained. It should be understood that in the embodiments disclosed in this specification, each data segment is processed as it is read. This is equivalent to initializing an empty first sub-index value and then continuously updating it statistically using different data segments until the updated first sub-index value is obtained.

[0074] It's important to understand that there can be one or more target graph metrics. This means that during the execution of a single data dump task, multiple graph metrics can be calculated simultaneously, each corresponding to a sub-metric value for the first slice. The following section uses a specific example of target graph metrics to illustrate how to stream-calculate the first sub-metric value corresponding to the first slice in a single data dump task.

[0075] I. Popularity of Knowledge

[0076] The aforementioned target graph metrics can include knowledge popularity, which indicates the N nodes with the most edges among the target type nodes, where N is a positive integer. The specific value can be manually set (e.g., N=10) or be a system default value (e.g., N=20). It's understood that graph data nodes generally involve different point types. For example, in a knowledge graph related to user consumption, point types include user nodes and product nodes. In this case, popular nodes under different point types can be counted. Generally, the more popular a node is, the more edges it is connected to, or in other words, the higher its degree. The aforementioned target type can be one of the point types involved in the graph data. It can be pre-set by staff, or each point type can be used as a target type by default.

[0077] Figure 5 This is a schematic diagram illustrating the process steps for calculating the first sub-index value of knowledge popularity as disclosed in the embodiments of this specification, wherein the illustrated calculation process includes the following steps:

[0078] Step S51: Based on the several data fragments related to the first node read sequentially, calculate the first degree of the first node, where the first node belongs to the target type; Step S52: Insert the first degree into the priority queue data structure corresponding to the target type, so that the degree inserted into the priority queue data structure dynamically eliminates the degree of nodes outside the top N positions; Step S53: In response to the completion of reading the first piece of data, use the N node degrees stored in the priority queue data structure as the first sub-index value of the knowledge heat.

[0079] Regarding step S51 above, it should be noted that "several" in the text refers to one or more. In addition, when there are multiple data segments, the data segments are not accumulated and then processed. Instead, they are processed one by one, and the first degree of the first node is updated by counting.

[0080] Next, the adaptive execution method of step S51 will be described, taking into account the exemplary description of the graph data partitioning mechanism and storage method of the graph data warehouse.

[0081] For example, the K fragments, including the first fragment, are obtained based on the point splitting mechanism. Multiple nodes in the first fragment can be used as center nodes to obtain multiple one-hop subgraphs corresponding to the multiple nodes. The data of the first fragment is stored according to the one-hop subgraphs. The data of the one-hop subgraphs includes the point data of the center node and the edge data of the one-hop connection edge connected to the center node.

[0082] Figure 6 This is a schematic diagram of the storage structure for one-hop subgraph data disclosed in the embodiments of this specification. It illustrates that the attribute data of the central node is stored in the first storage area of ​​the storage structure, and the one-hop edge data of the central node is stored in several second storage areas. It should be understood that... Figure 6 This is just one possible storage structure, and it is not the only one in reality.

[0083] For example, the attribute data of the central node includes:

[0084] User 701 <Occupation> "Student"

[0085] User 701 <Location> "Beijing"

[0086] User 701 <Interests and Hobbies> "Music"

[0087] The one-hop edge data of the central node includes:

[0088] [User 701, visited website 123]

[0089] [User 701, Traveling, Attractions 456]

[0090] [User 701 purchased item 789]

[0091] Furthermore, in one example, edge data can be stored in each second storage area according to edge type. Specifically, based on the upper limit of the storage space of the second storage area, dense relationships in the one-hop subgraph (e.g., Figure 6 The edge data corresponding to the purchase relationship shown in the diagram is split and stored separately in the second storage area, thus separating the sparse relationships (e.g., purchase relationships). Figure 6 The edge data corresponding to the access and outbound relationships shown in the diagram are merged and stored in the second storage area.

[0092] On the other hand, in one example, it can be designed so that the point attributes of each node in the first shard are stored only once; specifically, only in... Figure 6 The first storage area, as shown in the diagram, stores point attribute data, and only stores the point attribute data of the center node. The second storage area does not store point attribute data.

[0093] Furthermore, considering the presence of hotspots in the first segment, which have a massive number of connected edges (e.g., 10 million), multiple storage structures are needed to store the one-hop subgraph data corresponding to these hotspots, primarily edge data. To ensure that point attribute data is stored only once, the point attribute data of the central node can be stored in the first storage structure among the multiple storage structures. Instead of storing this point attribute data in other storage structures, predetermined indicators (such as placeholders) can be stored to indicate that the node information has not changed. For example, the predetermined indicators can be stored in the first storage area of ​​other storage structures.

[0094] The above describes the storage method of the first piece of data in the map data warehouse. Next, the adaptive execution method of step S51 will be described.

[0095] In one implementation, firstly, in response to reading a first data segment, a counter value is set for a first node, wherein the first data segment includes first attribute data of the first node, and a first number of edge information following the first attribute data as the segment end; wherein the counter value is the first number. Next, in response to reading at least one second data segment, wherein each second data segment includes a predetermined indicator and a second number of edge information, each second number is added to the aforementioned counter value, wherein the predetermined indicator indicates that the node information has not changed. Then, in response to reading attribute data of a second node (different from the first node) in a data segment, the current counter value is used as the first degree of the first node.

[0096] It's important to understand that in a graph data warehouse, if the first data segment is stored as a one-hop subgraph and the node's attribute data is stored only once, then reading the attribute data of another node indicates that the subgraph data of the previous node has been completely read. For example, a data segment may contain one or more... Figure 6 The storage structure is shown in the diagram.

[0097] In another implementation scenario, in response to reading a first data fragment, wherein the data entries include at least, in sequence, attribute data of a first node, edge information of a first number, and attribute data of a second node, the first number is determined as a first degree.

[0098] The above Figure 5 The execution method of step S51 shown is illustrated by way of example. Further, by executing steps S52 and S53, the degree of nodes other than the top N is dynamically eliminated using the priority queue data structure, so that the N node degrees stored in the priority queue data structure after the first shard data is read are used as the first sub-index value of knowledge heat.

[0099] Based on the above, during the execution of the first dump task, streaming computation can be performed on the read data fragments to obtain the first sub-index value of the knowledge popularity index.

[0100] II. Inbound / Outbound Edge Mounting Rate

[0101] The aforementioned target graph metrics can include outgoing / incoming edge mounting rate, where the outgoing edge mounting rate is defined as the ratio of the number of origin instances to the total number of origin type instances, given an edge type. It should be understood that the edge type simultaneously defines the origin type, edge relationship, and destination type, where the total number of origin type instances refers to the total number of node instances of the origin type corresponding to that edge type.

[0102] Similarly, the ingress edge mounting rate is defined as the ratio of the number of endpoint instances to the total number of endpoint type instances for a given edge type, where the total number of endpoint type instances refers to the total number of node instances with the endpoint type corresponding to the edge type.

[0103] Figure 7 This is a schematic diagram illustrating the process steps for calculating the first sub-index value of the outbound / inbound edge mounting rate as disclosed in the embodiments of this specification. The schematic calculation process includes the following steps:

[0104] Step S71: Using the data fragment, update the deduplication start / end point count of the target edge type, and update the node count of the start / end point type corresponding to the target edge type. It should be understood that the target edge type can be a certain edge type involved in the first data segment, or each edge type involved in the first data segment can be used as the target edge type.

[0105] For example, for a given edge type, the start / endpoint identifiers of the connecting edges of that type in the current data segment are inserted into a probabilistic data structure of HyperLogLog or HyperLogLog plusplus to obtain the updated deduplicated start / endpoint counts corresponding to that edge type. In this way, efficient deduplication counting can be achieved by tolerating a small amount of statistical bias.

[0106] Step S72: In response to the completion of reading the first shard data, the first sub-index value of the outgoing edge mounting rate is obtained based on the deduplication start point count of the target edge type and the node count of the corresponding start point type, and / or the first sub-index value of the incoming edge mounting rate is obtained based on the deduplication end point count and the node count of the corresponding end point type.

[0107] Based on the above, during the execution of the first dump task, streaming computation can be performed on the read data segments to obtain the first sub-index value of the outgoing edge load rate and / or the first sub-index value of the incoming edge load rate.

[0108] III. Invalid Edge Indicator

[0109] The aforementioned target graph metrics may include invalid edge metrics, which indicate the number of connected edges whose attribute data at the start or end point is empty.

[0110] Figure 8 This is a schematic diagram illustrating the process steps for calculating the first sub-index value of the invalid edge index as disclosed in the embodiments of this specification. The schematic calculation process includes the following steps:

[0111] Step S81: Update the count of invalid edges based on whether the point attribute data of the start or end point of each connection edge in the data segment is empty.

[0112] For example, in the graph data warehouse, the first piece of data is stored as a one-hop subgraph and the attribute data of the node is stored only once. In this case, if it is determined that the attribute data of a certain node is empty, the connecting edges connected to it can be classified as invalid edges, thereby updating the count of invalid edges.

[0113] Step S82: In response to the completion of reading the first shard data, determine the first sub-index value of the invalid edge index based on the updated invalid edge count.

[0114] In one embodiment, the updated invalid edge count can be directly determined as the first sub-index value of the invalid edge index. In another embodiment, considering that some statistical methods used in step S81 when counting invalid edges may repeatedly count invalid edges whose start and end attribute data are both empty, a method for deduplicating such invalid edges is proposed.

[0115] In a specific embodiment, after the first dump task is completed, the point table and edge table corresponding to the first shard are obtained. At this time, the point table and edge table can be joined by performing a join operation. For example, by designing a corresponding Structured Query Language (SQL) statement, invalid edges that do not exist at the start and end points can be obtained. Then, the invalid edge count obtained in step S81 is subtracted from the number of invalid edges obtained through the join operation, which is used as the first sub-index value of the invalid edge index.

[0116] Based on the above, streaming computation can be performed on the read data fragments during the execution of the first dump task to obtain the first sub-index value of the invalid edge index.

[0117] IV. Degree Distribution Index

[0118] The aforementioned target graph metrics may include degree distribution metrics, which indicate the number of node instances corresponding to different degrees for a given node type.

[0119] Figure 9 This is a schematic diagram illustrating the process steps for calculating the first sub-index value of the degree distribution index as disclosed in the embodiments of this specification. The schematic calculation process includes the following steps:

[0120] Step S91: Update the degree distribution corresponding to the target point type based on the degree of each node with the target point type determined based on the current data segment.

[0121] In implementation method A, to further improve computational efficiency, a sparse threshold and a dense threshold are pre-defined for the degree of the target point type, forming a degree threshold window. Therefore, if a data segment is parsed and contains several first nodes of the target point type, and the several first degrees corresponding to these first nodes are within the degree threshold window, then these first degrees are inserted into the CountMinSketch data structure to obtain the updated degree distribution corresponding to the target point type; and / or, if a data segment is parsed and contains several second nodes of the target point type, and the several second degrees corresponding to these second nodes are outside the degree threshold window, then the several second degrees are used to update the multidimensional degree-count tuple. It can be understood that "multidimensional" refers to having multiple tuples, with a degree-count tuple established for each degree outside the degree threshold window.

[0122] Furthermore, in a specific embodiment, considering that the CountMinSketch data structure cannot mark degrees, but only counts the number of node instances for each degree, a Bitmap data structure can be introduced to process the aforementioned first degrees, thereby updating the bitmap corresponding to this point type for marking different degrees. It can be understood that, to determine the number of node instances corresponding to a certain degree, the number of node instances marked in the bitmap can be hashed, and then the hash value can be used as an index to locate the corresponding count value in the CountMinSketch data structure; this count value is the number of node instances for that certain degree.

[0123] In implementation method B, the parsed data segment contains multiple nodes of the target point type and the degree of each node. Therefore, the parsed node degrees can be used to update multiple degree-count tuples corresponding to the target point type.

[0124] Step S92: In response to the completion of reading the first piece of data, the updated degree distribution corresponding to the target point type is assigned to the first sub-index value of the degree distribution index.

[0125] Corresponding to Implementation A above, in this step, the updated bitmap, degree distribution, and multi-dimensional number-count tuples under the target point type can be collectively classified into the first sub-index value of the degree distribution index. In another embodiment, the updated bitmap, degree distribution, and multi-dimensional number-count tuples under the target point type can be uniformly integrated into a degree-count tuple, and then classified into the first sub-index value of the degree distribution index.

[0126] Corresponding to Implementation B above, in this step, multiple degree-count pairs under the target point type can be classified as the first sub-index value of the degree distribution index.

[0127] Based on the above, streaming computation can be performed on the read data fragments during the execution of the first dump task to obtain the first sub-index value of the degree distribution index.

[0128] V. Attribute Coverage

[0129] The aforementioned target graph metrics may include attribute coverage, which indicates the proportion of node instances whose point attributes are not empty for a given node type.

[0130] Figure 10 This is a schematic diagram illustrating the process steps for calculating the first sub-index value of attribute coverage as disclosed in the embodiments of this specification, wherein the schematic calculation process includes the following steps:

[0131] Step S101: Based on the read current data fragment, update the attribute distribution and total number of nodes corresponding to the target point type, wherein the attribute distribution includes the number of nodes whose attribute values ​​are not empty for each attribute under the target point type.

[0132] For example, suppose the graph data of a one-hop subgraph is stored in the graph data warehouse as vertices and edges. For instance, see [link to relevant documentation]. Figure 5 The illustrated storage structure allows for updating the attribute distribution and total number of nodes for the target point type by scanning only the point attribute data within the data segment during streaming computation, without needing to scan the edge data. This effectively reduces computational load. It can be understood that this applies not only to the first sub-index value of attribute coverage, but also to any graph index related only to points, by effectively reducing computational load by scanning only the point attribute data within the data segment.

[0133] In step S102, in response to the completion of reading the first shard data, the updated attribute distribution and total number of nodes are assigned as the first sub-index value of attribute coverage.

[0134] Based on the above, streaming computation can be performed on the read data fragments during the execution of the first dump task to obtain the first sub-index value of attribute coverage.

[0135] The above examples, including knowledge popularity, inbound / outbound edge mounting rate, invalid edge index, degree distribution index, and attribute coverage, illustrate how streaming computation is performed on data segments during the execution of the first data dump task to obtain the first sub-index value of the target graph index corresponding to the first fragment. It's understandable that there can be many more instances of target graph indices; for example, they can be customized by staff according to business needs, and are not limited to the indices listed in this article.

[0136] Back Figure 4In step S410, in addition to calculating the first sub-index value, the format conversion of the data fragment is also involved, such as dumping it into an Open Data Processing Service (ODPS) table, which will not be elaborated here.

[0137] It should be noted that the first sub-index value calculated in step S410 corresponding to the first shard is used to merge or aggregate with the sub-index values ​​corresponding to other shards in the K shards to obtain the target index value. This aggregation process can be performed by the first device. Figure 4 The method shown may further include step S420, which aggregates the first sub-index value and the K-1 sub-index values ​​of the target map index corresponding to the other K-1 segments to obtain the target index value.

[0138] The following description of step S420 will also use specific examples of graph indicators to illustrate the execution process.

[0139] In one embodiment, the target graph index includes knowledge popularity. In this case, the execution of step S420 may include: based on the first sub-index value of knowledge popularity and the other K-1 sub-index values ​​of knowledge popularity determined in the other K-1 data dumping tasks, the degree of the top N nodes in K sets is accumulated and sorted by node to obtain the N nodes corresponding to the final degree of the top N nodes, which are used as the target index value of knowledge popularity.

[0140] In another embodiment, the target graph index includes the outbound / inbound edge mounting rate. In this case, the execution of step S420 may include: based on the first sub-index value of the outbound / inbound edge mounting rate and other sub-index values ​​of the outbound / inbound edge mounting rate determined in the other K-1 data dumping tasks, summing the deduplication start point count and the corresponding start point node count for the target edge type and calculating the ratio, which is used as the target index value of the outbound edge mounting rate; and / or summing the deduplication end point count and the corresponding end point node count for the target edge type and calculating the ratio, which is used as the target index value of the inbound edge mounting rate.

[0141] In another embodiment, the target graph index includes an invalid edge index. For example, the K fragments are obtained based on a point splitting mechanism. Based on this, the execution of step S420 may include: summing the first sub-index value of the invalid edge index and the other K-1 sub-index values ​​of the invalid edge index determined in the other K-1 data dumping tasks to obtain the target index value of the invalid edge index.

[0142] In another embodiment, the target map index includes a degree distribution index. In this case, the execution of step S420 may include: aggregating the first sub-index value of the degree distribution index and the other K-1 sub-index values ​​of the degree distribution index determined in the other K-1 data dumping tasks according to different degrees under the target point type to obtain the target index value of the degree distribution index.

[0143] In another embodiment, the target map index attribute coverage, at this time, the execution of step S420 may include sub-steps S421 and S422.

[0144] In sub-step S421, the attribute coverage is aggregated based on the first sub-index value and the other sub-index values ​​of attribute coverage determined in the other K-1 data dump tasks to obtain the aggregated attribute distribution and the total number of aggregated nodes corresponding to the target point type.

[0145] In sub-step S422, for each attribute in the aggregated attribute distribution corresponding to the target point type, the number of nodes with non-empty attribute values ​​under this attribute is divided by the number of aggregated nodes corresponding to the target point type to obtain the target attribute coverage rate of this attribute.

[0146] In one embodiment, considering the difficulty in statistically analyzing point attributes with zero coverage, it is advisable to use the underlying schema service, or the metadata used when constructing the graph, to determine this.

[0147] Specifically, step S422 may further include: using the standard point attribute entries returned by the schema service to match all attributes in the above aggregated attribute distribution; if it is determined that the matching result of a certain type of standard point attribute relative to the aggregated attribute distribution is unsuccessful, the attribute coverage rate of that certain type of standard point attribute is 0 and is added as the target attribute coverage rate.

[0148] The above examples, including knowledge popularity, inbound / outbound edge mounting rate, invalid edge index, degree distribution index, and attribute coverage, illustrate the process of aggregating sub-index values ​​from various data shards to obtain target index values.

[0149] It is understood that the above aggregation process can also be performed by other devices besides the first device. In this case, the first device can send the first sub-index value to other devices so that the other devices can perform the aggregation of the sub-index. The aggregation process can be found in the description of step S420, which will not be repeated here.

[0150] It should also be noted that all embodiments combined in describing steps S410 and S420 can be executed at least under the premise that K segments are obtained based on the point-splitting mechanism.

[0151] In summary, the graph index calculation method for graph data disclosed in the embodiments of this specification utilizes a reused dump link to perform streaming calculations on the sub-index values ​​of the graph index within a dump task targeting a single graph data fragment. These sub-index values ​​are then merged to obtain the complete index value. This effectively reduces the computational load and time required for graph index calculations, significantly improving computational efficiency.

[0152] Corresponding to the above-mentioned method for calculating spectral indices, this specification also discloses a device for calculating spectral indices. Figure 11 This is a schematic diagram of the structure of the graph data spectral index calculation device disclosed in the embodiments of this specification, wherein the graph data is divided into K segments and stored in a graph data warehouse; the device is integrated into a first device. Figure 11 As shown, the device 1100 includes:

[0153] The dump module 1110 is configured to execute a first data dump task in response to the triggering of a dump condition. The first data dump task is used to dump data for the first fragment among the K fragments.

[0154] The index calculation unit 1111 included in the dump module 1110 is configured to: in the process of reading the first fragment data corresponding to the first fragment from the graph data warehouse, perform streaming statistical calculations related to nodes and / or edges on the read data fragments to obtain the first sub-index value of the target graph index corresponding to the first fragment; the first sub-index value is used to aggregate with the sub-index values ​​corresponding to other fragments to obtain the target index value.

[0155] In one embodiment, the dump module 1110 is specifically configured to execute the first data dump task in parallel with other K-1 data dump tasks.

[0156] In one embodiment, the K segments are obtained based on a point-splitting mechanism. The first segment includes multiple one-hop subgraphs, wherein the data of each one-hop subgraph includes the point data of the central node and the edge data of the connecting edges connected to the central node.

[0157] In one embodiment, the target graph index includes knowledge popularity, which indicates the N nodes with the most edges among the target type nodes; wherein, the index calculation unit 1111 is specifically configured to: calculate the first degree of the first node based on several data fragments related to the first node read sequentially, wherein the first node belongs to the target type; insert the first degree into a priority queue data structure corresponding to the target type, so that the degree inserted into the priority queue data structure dynamically eliminates the degree of nodes outside the top N positions; in response to the completion of reading the first piece of data, use the N node degrees stored in the priority queue data structure as the first sub-index value of the knowledge popularity.

[0158] Furthermore, in a specific embodiment, the index calculation unit 1111 is specifically configured to calculate the first degree of the first node based on several data segments related to the first node read sequentially, including:

[0159] In response to reading a first data segment, a counter value is set for a first node; wherein the first data segment includes first attribute data of the first node, and a first number of edge information following the first attribute data as the segment end; the counter value is the first number. In response to reading at least one second data segment, wherein each second data segment includes a predetermined indicator and a second number of edge information, the respective second numbers are added to the counter value; wherein the predetermined indicator indicates that the node information has not changed. In response to reading attribute data of a second node in a data segment, the current counter value is used as the first degree.

[0160] In another specific embodiment, the index calculation unit 1111 is specifically configured to calculate the first degree of the first node based on a plurality of data segments related to the first node read sequentially, including: in response to reading the first data segment, wherein the data entries therein at least sequentially include the attribute data of the first node, the edge information of a first number, and the attribute data of the second node, determining the first number as the first degree.

[0161] On the other hand, in one embodiment, the device 1100 further includes: an index value merging module 1120, configured to accumulate and sort the degree values ​​of the top N nodes in K sets according to the first sub-index value of the knowledge popularity and other sub-index values ​​of the knowledge popularity determined in the other K-1 data dumping tasks, to obtain N nodes corresponding to the final top N node degrees, which are used as the target index value of the knowledge popularity.

[0162] In one embodiment, the target graph index includes outbound / inbound edge mounting rate; the index calculation unit 1111 is specifically configured to: update the deduplication start / end count of the target edge type using data fragments, and update the node count of the start / end point type corresponding to the target edge type; in response to the completion of reading the first fragment data, obtain a first sub-index value of the outbound edge mounting rate based on the deduplication start count of the target edge type and the node count of the corresponding start point type, and / or obtain a first sub-index value of the inbound edge mounting rate based on the deduplication end count of the target edge type and the node count of the corresponding end point type.

[0163] Furthermore, in a specific embodiment, the index calculation unit 1111 is specifically configured to use data fragments to update the deduplication start / end point count of the target edge type, including: inserting the start / end point identifier of the connecting edge with the target edge type in the data fragment into a probability data structure of HyperLogLog or HyperLogLog plusplus to obtain the updated deduplication start / end point count corresponding to the target edge type.

[0164] On the other hand, the device 1100 further includes an index value merging module 1120, configured to: based on the first sub-index value of the outbound / inbound edge loading rate and other sub-index values ​​of the outbound / inbound edge loading rate determined in the other K-1 data dumping tasks, accumulate the deduplication start point count and the corresponding start point node count of the target edge type and calculate the ratio as the target index value of the outbound edge loading rate, and / or accumulate the deduplication end point count and the corresponding end point node count of the target edge type and calculate the ratio as the target index value of the inbound edge loading rate.

[0165] In one embodiment, the target graph index includes an invalid edge index, which indicates the number of invalid edges; the index calculation unit 1111 is specifically configured to: update the count of invalid edges based on whether the point attribute data of the start or end point of each connecting edge in the data segment is empty; in response to the completion of reading the first fragment data, determine the first sub-index value of the invalid edge index based on the updated invalid edge count.

[0166] In a specific embodiment, the K shards are obtained based on a point-splitting mechanism; the device 1100 further includes an index value merging module 1120, configured to: sum the first sub-index value of the invalid edge index and the other sub-index values ​​of the invalid edge index determined in the other K-1 data dumping tasks to obtain the target index value of the invalid edge index.

[0167] In one embodiment, the target map index includes a degree distribution index; wherein, the index calculation unit 1111 is specifically configured to: update the degree distribution corresponding to the target point type based on the degree of each node with the target point type determined based on the data fragment; and in response to the completion of reading the first fragment data, assign the updated degree distribution corresponding to the target point type to the first sub-index value of the degree distribution index.

[0168] In a specific embodiment, the index calculation unit 1111 is specifically configured to update the degree distribution corresponding to the target point type based on the degree of each node with the target point type determined based on the data fragment, including:

[0169] Parsing reveals that the data segment contains several first nodes of the target point type, and that the first degrees corresponding to these first nodes are within a preset degree threshold window; these first degrees are then inserted into the CountMinSketch data structure to obtain the updated degree distribution corresponding to the target point type. And / or,

[0170] Parse the data segment to find that there are several second nodes with the target point type, and that several second degrees corresponding to these second nodes are outside the degree threshold window; use these second degrees to update the multidimensional degree-count tuple.

[0171] On the other hand, in a specific embodiment, the index calculation unit 1111 is further configured to: process the plurality of first degrees using a Bitmap data structure to update the bitmap corresponding to the target point type for marking different degrees.

[0172] In one embodiment, the device 1100 further includes an index value merging module 1120, configured to: aggregate the first sub-index value of the degree distribution index and other sub-index values ​​of the degree distribution index determined in the other K-1 data dumping tasks according to different degrees under the target point type to obtain the target index value of the degree distribution index.

[0173] In one embodiment, the target map features include attribute coverage; the index calculation unit 1111 is specifically configured to: update the attribute distribution and total number of nodes corresponding to the target point type according to the data fragment, wherein the attribute distribution includes the number of nodes whose attribute values ​​are not empty for each attribute under the target point type; in response to the completion of reading the first fragment data, assign the updated attribute distribution and total number of nodes to the first sub-index value of the attribute coverage.

[0174] In one embodiment, the device 1100 further includes an index value merging module 1120, configured to: aggregate the first sub-index value of the attribute coverage and other sub-index values ​​of the attribute coverage determined in the other K-1 data dumping tasks to obtain the aggregated attribute distribution and the total number of aggregated nodes corresponding to the target point type; for each attribute in the aggregated attribute distribution corresponding to the target point type, divide the number of nodes with non-empty attribute values ​​under this attribute by the total number of aggregated nodes corresponding to the target point type to obtain the target attribute coverage of the target attribute.

[0175] According to another embodiment, a computer-readable storage medium is also provided, on which a computer program is stored, which, when executed in a computer, causes the computer to perform a combination Figure 2 The method described.

[0176] According to another embodiment, a computing device is also provided, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, it implements a combination... Figure 2 The method described.

[0177] Those skilled in the art will recognize that, in one or more of the examples above, the functions described in this invention can be implemented using hardware, software, firmware, or any combination thereof. When implemented in software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or code on a computer-readable medium.

[0178] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made on the basis of the technical solution of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for calculating a graph index of graph data, wherein the graph data is divided into K fragments and stored in a graph data warehouse; the method is executed by a first device, comprising: In response to the triggering of a dump condition, a first data dump task is executed, which is used to dump data for the first fragment among the K fragments; wherein, executing the first data dump task includes: During the process of reading the first fragment data corresponding to the first fragment from the graph data warehouse, streaming statistical calculations related to nodes and / or edges are performed on the read data fragments to obtain the first sub-index value of the target graph index corresponding to the first fragment; the first sub-index value is used to aggregate the sub-index values ​​corresponding to other fragments to obtain the target index value.

2. The method according to claim 1, wherein, Executing the first data dump task includes: The first data dump task is executed in parallel with the other K-1 data dump tasks.

3. The method according to claim 1, wherein, The K segments are obtained based on a point-splitting mechanism. The first segment includes multiple one-hop subgraphs, where the data of each one-hop subgraph includes the point data of the central node and the edge data of the connecting edges connected to the central node.

4. The method according to claim 1, wherein, The target graph index includes knowledge popularity, which indicates the N nodes with the most edges among the target type nodes; wherein, streaming statistical calculations related to nodes and / or edges are performed on the read data segments to obtain the first sub-index value of the target graph index corresponding to the first segment, including: Based on several data segments related to the first node read sequentially, the first degree of the first node is calculated, wherein the first node belongs to the target type; The first degree is inserted into the priority queue data structure corresponding to the target type, so that the degree inserted into the priority queue data structure dynamically eliminates the degree of nodes that are not in the first N positions. In response to the completion of reading the first shard of data, the N node degrees stored in the priority queue data structure are used as the first sub-index value of the knowledge popularity.

5. The method according to claim 4, wherein, Based on the several data segments related to the first node read sequentially, the first degree of the first node is calculated, including: In response to reading a first data segment, a counter value is set for the first node; wherein the first data segment includes first attribute data of the first node, and a first number of edge information following the first attribute data as the end of the segment; the counter value is the first number; In response to reading at least one second data segment, wherein each second data segment includes a predetermined indicator and a second number of edge information, the respective second numbers are accumulated to the count value; wherein the predetermined indicator indicates that the node information has not changed; In response to reading attribute data of a second node in a data segment, the current count value of the counter is used as the first degree.

6. The method according to claim 4, wherein, Based on the several data segments related to the first node read sequentially, the first degree of the first node is calculated, including: In response to reading a first data segment, wherein the data entries include at least the attribute data of a first node, the edge information of a first number, and the attribute data of a second node in sequence, the first number is determined as a first degree.

7. The method according to claim 4, further comprising: Based on the first sub-index value of the knowledge popularity and the other sub-index values ​​of the knowledge popularity determined in the other K-1 data dumping tasks, the degree of the top N nodes in the K sets is accumulated and sorted by node to obtain the N nodes corresponding to the final top N node degrees, which are used as the target index value of the knowledge popularity.

8. The method according to claim 1, wherein, The target graph index includes the outgoing / incoming edge mounting rate; wherein, streaming statistical calculations related to nodes and / or edges are performed on the read data segments to obtain the first sub-index value of the target graph index corresponding to the first segment, including: Using data fragments, update the deduplication start / end point counts of the target edge type, and update the node counts of the start / end point types corresponding to the target edge type; In response to the completion of reading the first shard data, the first sub-index value of the outgoing edge mounting rate is obtained based on the deduplication starting point count of the target edge type and the node count of the corresponding starting point type, and / or the first sub-index value of the incoming edge mounting rate is obtained based on the deduplication ending point count of the target edge type and the node count of the corresponding ending point type.

9. The method according to claim 8, wherein, Using data fragments, update the deduplication start / end counts of the target edge type, including: Insert the start / endpoint identifiers of the connecting edges with the target edge type in the data segment into the probability data structure of HyperLogLog or HyperLogLog plusplus to obtain the updated deduplicated start / endpoint counts corresponding to the target edge type.

10. The method of claim 8, further comprising: Based on the first sub-index value of the outbound / inbound edge loading rate, and the other sub-index values ​​of the outbound / inbound edge loading rate determined in the other K-1 data dumping tasks, the ratio of the deduplication start point count and the corresponding start point node count of the target edge type is calculated as the target index value of the outbound edge loading rate, and / or the ratio of the deduplication end point count and the corresponding end point node count of the target edge type is calculated as the target index value of the inbound edge loading rate.

11. The method according to claim 1, wherein, The target graph index includes an invalid edge index, which indicates the number of invalid edges; wherein, streaming statistical calculations related to nodes and / or edges are performed on the read data segments to obtain a first sub-index value of the target graph index corresponding to the first segment, including: Update the count of invalid edges based on whether the point attribute data of the start or end point of each connecting edge in the data segment is empty; In response to the completion of reading the first shard data, the first sub-index value of the invalid edge index is determined based on the updated invalid edge count.

12. The method according to claim 11, wherein, The K fragments are obtained based on a point-based partitioning mechanism; the method further includes: The target value of the invalid edge index is obtained by summing the first sub-index value of the invalid edge index and the other sub-index values ​​of the invalid edge index determined in the other K-1 data dumping tasks.

13. The method according to claim 1, wherein, The target graph index includes a degree distribution index; wherein, based on the read data segments, streaming computation related to nodes and / or edges is performed to obtain the first sub-index value of the target graph index corresponding to the first piece, including: Update the degree distribution corresponding to the target point type based on the degree of each node with the target point type determined based on the data fragment; In response to the completion of reading the first piece of data, the updated degree distribution corresponding to the target point type is assigned to the first sub-index value of the degree distribution index.

14. The method according to claim 13, wherein, Based on the degree of each node with the target point type determined from the data fragment, update the degree distribution corresponding to the target point type, including: Parse the data segment to find several first nodes with the target point type, and the several first degrees corresponding to these first nodes are located within a preset degree threshold window; insert these several first degrees into the CountMinSketch data structure to obtain the updated degree distribution corresponding to the target point type; and / or, Parse the data segment to find that there are several second nodes with the target point type, and that several second degrees corresponding to these second nodes are outside the degree threshold window; use these second degrees to update the multidimensional degree-count tuple.

15. The method of claim 14, further comprising: The first degree values ​​are processed using a Bitmap data structure to update the bitmap corresponding to the target point type for marking different degree values.

16. The method of claim 13, further comprising: Based on the first sub-index value of the degree distribution index and the other sub-index values ​​of the degree distribution index determined in the other K-1 data dumping tasks, the index is aggregated according to different degrees under the target point type to obtain the target index value of the degree distribution index.

17. The method according to claim 1, wherein, The target graph features include attribute coverage; wherein, based on the read data segments, streaming computation related to nodes and / or edges is performed to obtain the first sub-index value of the target graph index corresponding to the first segment, including: Based on the data fragment, update the attribute distribution and total number of nodes corresponding to the target point type, wherein the attribute distribution includes the number of nodes whose attribute values ​​are not empty for each attribute under the target point type; In response to the completion of reading the first shard data, the updated attribute distribution and total number of nodes are assigned as the first sub-index value of the attribute coverage.

18. The method of claim 17, further comprising: Aggregate the first sub-index value of the attribute coverage and the other sub-index values ​​of the attribute coverage determined in the other K-1 data dumping tasks to obtain the aggregated attribute distribution and the total number of aggregated nodes corresponding to the target point type. For each attribute in the aggregated attribute distribution corresponding to the target point type, the target attribute coverage rate is obtained by dividing the number of nodes with non-empty attribute values ​​under that attribute by the total number of aggregated nodes corresponding to the target point type.

19. A device for calculating map indexes of map data, wherein the map data is divided into K segments and stored in a map data warehouse; the device is integrated into a first device and includes: The dump module is configured to execute a first data dump task in response to the triggering of a dump condition. The first data dump task is used to dump data for the first of the K shards. The index calculation unit included in the dump module is configured to: perform streaming statistical calculations related to nodes and / or edges on the read data fragments during the process of reading the first fragment data corresponding to the first fragment from the graph data warehouse, and obtain the first sub-index value of the target graph index corresponding to the first fragment; the first sub-index value is used to aggregate the sub-index values ​​corresponding to other fragments to obtain the target index value.

20. A computer-readable storage medium having a computer program stored thereon, wherein, When the computer program is executed in the computer, it causes the computer to perform the method according to any one of claims 1-18.

21. A computing device comprising a memory and a processor, wherein, The memory stores executable code, and when the processor executes the executable code, it implements the method of any one of claims 1-18.

Images