A graph data processing method, device, electronic device and storage medium
By introducing the associated data and detailed data of the sub-graph in the graph data processing and processing incremental graph data, the problem of low computing efficiency of large-scale graph data in the prior art is solved, and fast and efficient sub-graph calculation and high concurrency processing are realized.
Patent Information
- Application Number
- CN202010842906.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-08-20
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2040-08-20
AI Technical Summary
When processing large graph data, real-time computing is difficult to process sub-graph traversal queries on super-large graphs, and offline computing gradually exceeds the computing capacity due to the correlation dependence of graphs, which cannot meet the high concurrency processing requirements.
By introducing the associated data and detailed data of the sub-graph, the new version of the target sub-graph is processed according to the incremental map data, and only the incremental data is processed to reduce the calculation amount and improve processing efficiency.
It realizes rapid sub-graph calculation, reduces data calculation amount, improves processing efficiency, meets users' needs for sub-graph calculation, and supports high concurrency processing.
Smart Images

Figure CN112000848B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer technology, and in particular to data processing and data storage technology, and specifically to a graph data processing method, device, electronic device and storage medium. Background Art
[0002] A graph database is a database built based on a graph relationship (not an image) model, and the data contained in it often includes point data and edge data. In the field of graph databases, there is often a need for subgraph calculations, especially in the case of calculating weakly connected subgraphs in graphs with large amounts of data, such as how many weakly connected subgraphs there are, whether multiple points belong to the same weakly connected subgraph, and what points and edges a weakly connected subgraph contains.
[0003] At present, real-time calculation and offline calculation are mainly used. The real-time calculation method is to traverse the input point data and edge data on demand on a graph with a large amount of data, and perform calculations in the form of a local graph. This method is difficult to handle subgraph traversal queries on very large graphs, and cannot achieve high concurrency processing. The offline calculation method is based on batch data, and offline data is pre-constructed as subgraph information and stored for subsequent queries. However, due to the correlation dependency of the graph, this method requires all data to be included in the recalculated subgraph each time. As time goes by, the amount of data calculated becomes larger and larger, even exceeding the computing power of the online environment. Summary of the invention
[0004] Embodiments of the present invention provide a graph data processing method, device, electronic device and storage medium, which reduce the amount of data calculation and improve the efficiency of graph data processing.
[0005] In a first aspect, an embodiment of the present invention provides a graph data processing method, the method comprising:
[0006] Determine the target subgraph and the newly added data of the target subgraph according to the incremental graph data;
[0007] Determining the incremental associated data of the target subgraph according to the midpoint data of the newly added data of the target subgraph;
[0008] Determining new version associated data of the target subgraph according to the incremental associated data and the original associated data of the target subgraph;
[0009] Determine new version detail data of the target subgraph according to the new version association data of the target subgraph and the edge data in the target subgraph;
[0010] The associated data includes the endpoint id and subgraph id; the detailed data includes the subgraph id, edge id, edge type, edge-associated endpoint id and endpoint type.
[0011] In a second aspect, an embodiment of the present invention further provides a graph data processing device, the device comprising:
[0012] A target data determination module, used to determine a target subgraph and newly added data of the target subgraph according to the incremental graph data;
[0013] An incremental associated data determination module, used to determine the incremental associated data of the target subgraph according to the midpoint data of the newly added data of the target subgraph;
[0014] A new associated data determination module, used to determine the new version associated data of the target subgraph according to the incremental associated data and the original associated data of the target subgraph;
[0015] A new detail data determination module, used to determine the new version detail data of the target subgraph according to the new version association data of the target subgraph and the edge data in the target subgraph;
[0016] The associated data includes the endpoint id and subgraph id; the detailed data includes the subgraph id, edge id, edge type, edge-associated endpoint id and endpoint type.
[0017] In a third aspect, an embodiment of the present invention further provides an electronic device, the electronic device comprising:
[0018] one or more processors;
[0019] A memory for storing one or more programs;
[0020] When the one or more programs are executed by the one or more processors, the one or more processors implement the graph data processing method as described in any embodiment of the present invention.
[0021] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the graph data processing method as described in any embodiment of the present invention.
[0022] The technical solution provided by the embodiment of the present invention lays the foundation for fast subgraph calculation by introducing the associated data and detailed data of the subgraph; at the same time, the incremental graph data is processed to obtain the new version associated data and new version detailed data of the target subgraph for subgraph calculation. Compared with the existing offline calculation method, this embodiment only processes the incremental graph data, which reduces the amount of data calculation, improves the data processing efficiency, and can meet the user's needs for subgraph calculation. BRIEF DESCRIPTION OF THE DRAWINGS
[0023] Figure 1 is a flowchart of a graph data processing method provided in Embodiment 1 of the present invention;
[0024] Figure 2 is a flowchart of a graph data processing method provided in Embodiment 2 of the present invention;
[0025] Figure 3 is a flowchart of a graph data processing method provided in Embodiment 3 of the present invention;
[0026] Figure 4 is a schematic diagram of a graph data processing process provided in Embodiment 4 of the present invention;
[0027] Figure 5 is a structural block diagram of a graph data processing device provided in Embodiment 5 of the present invention;
[0028] Figure 6 It is a structural schematic diagram of an electronic device provided in Embodiment 6 of the present invention. DETAILED DESCRIPTION
[0029] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.
[0030] Embodiment 1
[0031] Figure 1 This is a flowchart of a graph data processing method provided in the first embodiment of the present invention. This embodiment is applicable to the situation of how to process graph data, and is particularly applicable to the situation of how to efficiently process graph data to meet the user's needs for subgraph computing. The method can be executed by a graph data processing device, which can be implemented in software and / or hardware. Optionally, the device can be configured in a device that carries the graph data processing function, such as a server. See Figure 1 , the method may specifically include:
[0032] S110, determining the target subgraph and the newly added data of the target subgraph according to the incremental graph data.
[0033] It can be understood that a graph is a graph, which is composed of endpoints and edges. Correspondingly, graph data can include two types of edge data and point data. Point data can include endpoint id, endpoint attributes, and endpoint type, etc. Edge data can include two endpoint ids, edge type, edge id, and edge attributes associated with the edge. Among them, the endpoint id is an identifier used in graph data to uniquely represent the entity represented by the endpoint, such as a person's ID number; optionally, in order to facilitate subsequent calculations, queries, etc., the endpoint id and edge id in this embodiment can preferably be a fixed-length unique id assigned according to the set id allocation rules, such as a fixed-length unique id can be assigned to each endpoint and edge in sequence according to natural numbers. Endpoint types can include but are not limited to people, things, places, and objects; edge types are used to characterize the relationship between the two endpoints associated with the edge, which can include but are not limited to friendship, kinship, and affiliation. Further, in order to facilitate data storage, the edge type and point type in this embodiment can be fixed-length values encoded by a dictionary.
[0034] In this embodiment, the incremental graph data is the content added on the basis of the original graph data (also referred to as old graph data or existing graph data, etc.). The original graph data is the sum of graph data processed in all batches before the current batch processing. Optionally, the incremental graph data can be obtained from a path specified by the user. Furthermore, the point data and edge data in the incremental graph data can be stored in different paths, and then the point data and edge data can be obtained from the point data storage path and the edge data storage path, respectively.
[0035] In this embodiment, the subgraph may include a weakly connected subgraph, which is a connected graph, and any endpoint in the graph may not be fully connected to other endpoints; further, the number of endpoints in the subgraph may be one, or two or more. The target subgraph may be a subgraph included in the incremental graph data, and may specifically include a newly added subgraph due to the introduction of the incremental graph data, and / or a changed subgraph, the so-called changed subgraph is a subgraph after the connection structure of the existing subgraph is changed due to the introduction of the incremental graph data, for example, the existing subgraph includes a subgraph consisting of endpoints 5, 6 and 8, and there is an edge between endpoints 5 and 6 in the subgraph, and an edge between endpoint 8 and endpoint 5, and an edge is introduced in the incremental graph data, which is the edge between endpoint 8 and endpoint 4, and the changed subgraph is a subgraph consisting of endpoints 4, 5, 6 and 8. Optionally, the existing subgraph in this embodiment is a subgraph calculated based on the original graph data (further, a subgraph calculated based on all batches before the current batch). Furthermore, the existing subgraph may include an atomic graph and a change subgraph. The so-called atomic graph is an existing subgraph that is not affected by the incremental graph data, and the change subgraph is an existing subgraph that is affected by the incremental graph data (for example, the subgraph formed by the above-mentioned endpoints 5, 6 and 8 is a change subgraph).
[0036] Furthermore, this embodiment may provide a visual interactive interface, and the user may input relevant parameters according to the prompt information in the visual interface, such as the storage path of the incremental graph data (further including the storage path of the point data and the edge data in the incremental graph data) and the incremental calculation identifier, etc. Among them, the incremental calculation identifier is a mark used to indicate that the graph data is incrementally calculated, and correspondingly, there may also be a full calculation identifier used to indicate that the graph data is fully calculated.
[0037] Furthermore, the parameters such as the incremental calculation identifier input by the user, the point data storage path and the edge data storage path in the incremental graph data can be extracted from the visual interface, and the graph data can be incrementally calculated after the incremental calculation identifier is identified, that is, the process of S110 to S140 in this embodiment is executed. Specifically, the point data and edge data in the incremental graph data can be obtained from the point data storage path and the edge data storage path respectively, and then the target subgraph and the newly added data of the target subgraph can be determined based on the obtained data.
[0038] Furthermore, determining the target subgraph according to the incremental graph data may be: extracting the first point set from the point data of the incremental graph data; extracting the endpoints associated with the edge from the edge data of the incremental graph data as the second point set; optionally, since the same endpoint may be associated with two or more edges (for example, there is an edge between endpoint 1 and endpoint 2, there is an edge between endpoint 1 and endpoint 3, etc.), after extracting the endpoints associated with the edge from the edge data of the incremental graph data, the extracted endpoints may be deduplicated, and the deduplicated endpoints may be used as the second point set. Afterwards, the difference between the first point set and the second point set may be used as an isolated point set; since the incremental graph data is newly added data based on the original graph data, the isolated point set may be directly used as the newly added subgraph.
[0039] At the same time, a graph object can be constructed based on the edge data in the incremental graph data, and an open source graph computing framework (such as the Spark GraphX framework) can be used to perform subgraph calculations to obtain a first subgraph; a subgraph in the existing subgraph that has a common endpoint with the first subgraph (i.e., a changed subgraph) is used as the second subgraph, and the first subgraph and the second subgraph are merged to form a changed subgraph; further, if there is no subgraph in the existing subgraph that has a common endpoint with the first subgraph, the first subgraph can be directly used as the newly added subgraph.
[0040] Afterwards, the newly added subgraph and the changed subgraph can be used as the target subgraph. After determining the target subgraph, the newly added data of the target subgraph can also be determined. Furthermore, the data related to the target subgraph in the incremental graph data can be used as the newly added data of the target subgraph. In this embodiment, the newly added data is the newly added data. For example, for the target subgraph composed of the above-mentioned endpoints 4, 5, 6 and 8, the newly added data is the data of the edge between endpoint 8 and endpoint 4.
[0041] S120, determining the incremental associated data of the target subgraph according to the midpoint data of the newly added data of the target subgraph.
[0042] In this embodiment, the associated data can be stored in rows, and each row can include an endpoint id and a subgraph id, and the structure of each row can be expressed as (endpoint id, subgraph id). Optionally, the subgraph id can be the smallest endpoint id among the endpoints included in the subgraph, or it can also be the largest endpoint id among the endpoints included in the subgraph, etc. This embodiment and subsequent embodiments will be described using the subgraph id as the smallest endpoint id among the endpoints included in the subgraph as an example, but are not limited to this. The incremental associated data of the target subgraph is the associated data determined based on the newly added data of the target subgraph.
[0043] Specifically, after determining the target subgraph and the newly added data of the target subgraph, the incremental associated data of the target subgraph may be determined based on the associated data storage method and according to the midpoint data of the newly added data of the target subgraph.
[0044] Optionally, after determining the incremental associated data of the target subgraph, the incremental associated data of the target subgraph can be saved to a temporary space of the file system, wherein the file system of this embodiment can be an open source HDFS (Hadoop Distributed File System). It should be noted that in this embodiment, for each batch of incremental graph data, after the incremental associated data is obtained based on the incremental graph data, the incremental associated data is saved to a temporary space of the file system according to the calculation timestamp, which is convenient for subsequent queries and can meet the needs of data rollback and redo. For example, if the incremental associated data obtained based on a batch of incremental graph data is incorrect and needs to be rolled back, then the incremental associated data obtained by processing any batch of incremental graph data can be quickly located based on the calculation timestamp provided by this embodiment, without the need for large-scale data processing.
[0045] S130, determining new version associated data of the target subgraph according to the incremental associated data and the original associated data of the target subgraph.
[0046] In this embodiment, the original associated data of the target subgraph is the associated data of the target subgraph obtained based on the original graph data (further, the associated data related to the target subgraph obtained by integrating all batches processed before the current batch). For example, the target subgraph is composed of endpoints 5, 6, 8 and 10, and the subgraph composed of endpoints 6, 8 and 10 is a changed subgraph in the existing subgraph, and the associated data of the changed subgraph is the original associated data of the target subgraph. The new version associated data of the target subgraph may include the changed associated data of the target subgraph, for example, it may include the modified associated data of the target subgraph and / or the added associated data of the target subgraph; further, the new version associated data of the target subgraph may also include the unchanged associated data of the target subgraph, that is, the original associated data of the target subgraph.
[0047] Specifically, for each target subgraph, if the target subgraph is a newly added subgraph due to the introduction of incremental graph data, the target subgraph has no original associated data, and the incremental associated data of the target subgraph is the new version associated data of the target subgraph. If the target subgraph is a changed subgraph, the new version associated data of the target subgraph can be determined based on the endpoint id in the incremental associated data and the id of the target subgraph, and the endpoint id in the original associated data and the id of the target subgraph.
[0048] Optionally, after determining the new version associated data of the target subgraph, the number of subgraphs can be calculated based on the new version associated data of the target subgraph and the associated data of the existing subgraphs (which can further be the associated data of the atomic graph). For example, the subgraph id can be extracted from the new version associated data of the target subgraph obtained by the current batch calculation (such as the nth calculation) and the associated data of the existing subgraphs obtained by all batch calculations before the current batch (such as the first n-1 calculations) (further, if the new version associated data of the target subgraph includes unchanged associated data of the target subgraph, this can be the associated data of the atomic graph in the existing subgraph), and deduplication processing can be performed. After that, the number of subgraph ids after the deduplication processing can be counted, and the counted number of subgraph ids can be used as the number of subgraphs.
[0049] Furthermore, it is also possible to determine whether multiple endpoints belong to the same subgraph. For example, each endpoint id specified by the user can be used as an index to query the subgraph id associated with the endpoint id from the new version associated data of the target subgraph obtained by the current batch calculation (such as the nth calculation) and the associated data of the existing subgraphs obtained by all batch calculations before the current batch (such as the first n-1 calculations); then, it is possible to determine whether each endpoint belongs to the same subgraph based on the subgraph id associated with each endpoint id obtained.
[0050] S140, determining new version detail data of the target subgraph according to the new version association data of the target subgraph and the edge data in the target subgraph.
[0051] In this embodiment, the detail data can be stored by row, and each row can include subgraph id, edge id, edge type, first endpoint id, first endpoint type, second endpoint id, and second endpoint type, etc. The structure of each row can be expressed as (subgraph id, edge id, edge type, first endpoint id, first endpoint type, second endpoint id, second endpoint type), where the first endpoint and the second endpoint are the two endpoints associated with the edge.
[0052] Specifically, after determining the new version associated data of the target subgraph, an edge object can be constructed according to the edge data in the target subgraph, and the new version detailed data of the target subgraph can be determined according to the new version associated data of the target subgraph and the edge object. The structure of the edge object can be expressed as (edge id, edge type, first endpoint id, first endpoint type, second endpoint id, second endpoint type).
[0053] Furthermore, in this embodiment, after determining the incremental associated data of the target subgraph, an edge object can be constructed according to the edge data in the newly added data of the target subgraph; the incremental detail data of the target subgraph can be determined according to the incremental associated data of the target subgraph and the constructed edge object; and the new version detail data of the target subgraph can be determined according to the incremental detail data and the original detail data of the target subgraph. In this embodiment, the incremental detail data of the target subgraph can also be stored in the temporary space of the file system for subsequent query.
[0054] Optionally, after determining the new version detail data of the target subgraph, the subgraph detail data may be calculated. For example, the subgraph id specified by the user may be used as an index to query the detail data of the subgraph corresponding to the subgraph id from the new version detail data of the target subgraph obtained by the current batch calculation (such as the nth calculation) and the detail data of the existing subgraphs obtained by all batch calculations before the current batch (such as the first n-1 calculations).
[0055] It should be noted that the existing offline calculation method, as time goes by, the amount of data calculated increases; while this embodiment introduces the associated data and detailed data of the subgraph, and each batch can only process the incremental graph data, which reduces the amount of data calculation, improves data processing efficiency, and can meet the user's needs for subgraph calculation. In addition, this embodiment can also concurrently process multiple subgraph calculations of different users or the same user.
[0056] The technical solution of the embodiment of the present invention lays the foundation for fast subgraph calculation by introducing the associated data and detailed data of the subgraph; at the same time, the incremental graph data is processed to obtain the new version associated data and new version detailed data of the target subgraph for subgraph calculation. Compared with the existing offline calculation method, this embodiment only processes the incremental graph data, which reduces the amount of data calculation, improves the data processing efficiency, and can meet the user's needs for subgraph calculation.
[0057] Embodiment 2
[0058] Figure 2 is a flowchart of a graph data processing method provided in the second embodiment of the present invention. Based on the above embodiment, this embodiment provides a solution for determining incremental associated data and new version associated data. Figure 2 , the method may specifically include:
[0059] S210: Determine the target subgraph and the newly added data of the target subgraph according to the incremental graph data.
[0060] S220: Determine the first id of the target subgraph according to the endpoint id in the newly added data of the target subgraph.
[0061] Furthermore, for each target subgraph, if the target subgraph is a subgraph newly added due to the introduction of incremental graph data, and the number of endpoints included in the target subgraph is one, the id of the endpoint included in the target subgraph can be used as the first id of the target subgraph. If the target subgraph is a subgraph newly added due to the introduction of incremental graph data, and the number of endpoints included in the target subgraph is two or more, the minimum endpoint id can be selected from the various endpoint ids of the newly added data of the target subgraph as the first id of the target subgraph. If the target subgraph is a changed subgraph, the minimum endpoint id can be selected from the various endpoint ids of the newly added data of the target subgraph as the first id of the target subgraph.
[0062] S230: Determine the incremental associated data of the target subgraph according to the first id of the target subgraph and the endpoint id in the newly added data of the target subgraph.
[0063] Optionally, for each target subgraph, if the target subgraph includes one endpoint, the id of the endpoint included in the target subgraph can be associated with the first id of the target subgraph as the incremental associated data of the target subgraph; if the target subgraph includes two or more endpoints, the id of each endpoint included in the target subgraph can be associated with the first id of the target subgraph respectively, so as to obtain the incremental associated data of the target subgraph.
[0064] S240 , determining a target id of the target subgraph according to the first id of the target subgraph in the incremental association data and the second id of the target subgraph in the original association data.
[0065] Optionally, for each target subgraph, if the target subgraph is a newly added subgraph due to the introduction of incremental graph data, the target subgraph has no original associated data, and thus the first id of the target subgraph is the target id of the target subgraph.
[0066] If the target subgraph is a changed subgraph, the first id and the second id of the target subgraph can be compared. If the first id of the target subgraph is smaller than the second id, the first id of the target subgraph can be used as the target id of the target subgraph; if the first id of the target subgraph is greater than the second id, the second id of the target subgraph can be used as the target id of the target subgraph; if the first id of the target subgraph is equal to the second id, the second id or the first id can be used as the target id of the target subgraph.
[0067] S250 , determining new version associated data of the target subgraph according to the target id of the target subgraph and the endpoint id in the target subgraph.
[0068] In this embodiment, the new version associated data of the target subgraph may include modified associated data of the target subgraph and / or added associated data of the target subgraph.
[0069] Optionally, for each target subgraph, if the target subgraph is a subgraph newly added due to the introduction of incremental graph data, the incremental associated data of the target subgraph may be used as the new version associated data of the target subgraph.
[0070] If the target subgraph is a changed subgraph, and the first id of the target subgraph is greater than the second id of the target subgraph, that is, the second id of the target subgraph is the target id of the target subgraph, then the endpoint id in the incremental associated data of the target subgraph can be associated with the target id of the target subgraph, and the associated data can be added to the new version associated data of the target subgraph. Furthermore, the target id of the target subgraph can be used to replace the first id in the incremental associated data of the target subgraph, and the incremental associated data after the replacement process can be deduplicated (that is, the same parts as the original associated data are deleted), and then the incremental associated data after the deduplication process can be added as the associated data to the new version associated data of the target subgraph. For example, the target subgraph is composed of endpoints 5, 6, 8 and 10, and the original associated data of the target subgraph includes (5, 5), (8, 5) and (10, 5); the incremental associated data of the target subgraph includes (8, 6) and (6, 6), the first id of the target subgraph (i.e. 6) is greater than the second id of the target subgraph (i.e. 5), and thus the associated data added to the new version of the associated data of the target subgraph is (6, 5).
[0071] If the target subgraph is a changed subgraph, and the first id of the target subgraph is less than the second id of the target subgraph, that is, the first id of the target subgraph is the target id of the target subgraph, then the endpoint id of the target subgraph and the target id of the target subgraph can be associated as the new version of the associated data of the target subgraph. Furthermore, the target id of the target subgraph can be used to replace the second id in the original associated data of the target subgraph, and the original associated data after the replacement process can be used as the modified associated data in the new version of the associated data of the target subgraph; at the same time, the incremental associated data can be deduplicated (that is, the same parts as the original associated data after the replacement process are deleted), and then the incremental associated data after the deduplication process can be added as the associated data in the new version of the associated data of the target subgraph. For example, the target subgraph is composed of endpoints 4, 5, 6, 8 and 10, and the original associated data of the target subgraph includes (5, 5), (6, 5) (8, 5) and (10, 5); the incremental associated data of the target subgraph includes (6, 4) and (4, 4), the first id of the target subgraph (i.e. 4) is smaller than the second id of the target subgraph (i.e. 5), and thus the modified associated data in the new version associated data of the target subgraph includes (5, 4), (6, 4) (8, 4) and (10, 4); the added associated data in the new version associated data of the target subgraph includes (4, 4).
[0072] If the target subgraph is a changed subgraph, and the first id of the target subgraph is equal to the second id of the target subgraph, that is, the second id of the target subgraph or the first id is the target id of the target subgraph, then the incremental associated data can be deduplicated (that is, the same parts as the original associated data are deleted), and then the incremental associated data after deduplication can be added as the new version of the associated data of the target subgraph. For example, the target subgraph is composed of endpoints 5, 6, 8 and 10, and the original associated data of the target subgraph include (5, 5), (8, 5) and (10, 5); the incremental associated data of the target subgraph include (5, 5) and (6, 5), the first id of the target subgraph (that is, 5) is equal to the second id of the target subgraph (that is, 5), and then the associated data added to the new version of the associated data of the target subgraph include (6, 5).
[0073] S260 , determining new version detail data of the target subgraph according to the new version association data of the target subgraph and the edge data in the target subgraph.
[0074] Optionally, when the new version associated data of the target subgraph includes modified associated data of the target subgraph and / or added associated data of the target subgraph, the new version detail data of the target subgraph may also include modified detail data and / or added detail data of the target subgraph.
[0075] Optionally, according to the new version associated data of the target subgraph and the edge data in the target subgraph, determining the new version detail data of the target subgraph can be: determining the modified detail data in the new version detail data of the target subgraph according to the modified associated data of the target subgraph and the edge data in the original data of the target subgraph; determining the added detail data in the new version detail data of the target subgraph according to the added associated data of the target subgraph and the edge data in the newly added data of the target subgraph. The original data of the target subgraph in this embodiment is all the graph data in the target subgraph except the newly added data, or it can be said that the graph data related to the target subgraph in all batches before the current batch are integrated.
[0076] Specifically, an edge object can be constructed based on the edge data in the original data of the target subgraph; then, the modified detail data in the new version detail data of the target subgraph can be determined based on the modified associated data of the target subgraph and the constructed edge object. At the same time, an edge object can also be constructed based on the edge data in the newly added data of the target subgraph; then, the added detail data in the new version detail data of the target subgraph can be determined based on the edge object constructed in the added associated data of the target subgraph. This embodiment determines the new version detail data of the target subgraph based on the new version associated data of the target subgraph, providing a new idea for quickly and accurately determining the detail data of the subgraph.
[0077] The technical solution provided by the embodiment of the present invention determines the incremental associated data of the target subgraph according to the newly added data of the target subgraph; then, the new version associated data of the target subgraph can be obtained according to the first id of the target subgraph in the determined incremental associated data, the second id of the target subgraph in the original associated data, and the endpoint id in the target subgraph, and the new version detailed data can be obtained based on the new version associated data to perform subgraph calculation. Compared with the existing offline calculation method, this embodiment only processes incremental graph data, reduces the amount of data calculation, improves data processing efficiency, and can meet the user's needs for subgraph calculation.
[0078] Embodiment 3
[0079] Figure 3 is a flowchart of a graph data processing method provided in the third embodiment of the present invention. Based on the above embodiment, this embodiment adds the process of updating the associated query table and the detailed query. Figure 3 , the method may specifically include:
[0080] S310, determining a target subgraph and newly added data of the target subgraph according to the incremental graph data.
[0081] S320, determining the incremental associated data of the target subgraph according to the midpoint data of the newly added data of the target subgraph.
[0082] S330, determining new version associated data of the target subgraph according to the incremental associated data and the original associated data of the target subgraph.
[0083] S340 , determining new version detail data of the target subgraph according to the new version association data of the target subgraph and the edge data in the target subgraph.
[0084] S350: using the new version of the target subgraph's associated data, updating the associated query table including the associated data of the existing subgraph.
[0085] In order to query data with low latency, the data in the associated query table in this embodiment is stored in a column storage manner, and can further be stored in a column storage manner based on HBase. For example, Table 1 shows a data storage structure based on the HBase storage engine. Among them, the row key length is 16 bytes, and the endpoint id and subgraph id in the row key are both long integers with a length of 8 bytes.
[0086]
[0087] Table 1. Associative data storage structure based on HBase storage engine
[0088] Furthermore, after determining the new version of the associated data of the target subgraph, the new version of the associated data of the target subgraph can be formatted first, that is, the new version of the associated data of the target subgraph can be converted into query structure data; then, the data after the format conversion can be used to update the associated query table including the associated data of the existing subgraph. Furthermore, the new version of the associated data of the target subgraph can include the modified associated data and / or added associated data of the target subgraph, and the modified associated data after the format conversion can be used to replace the related data in the associated query table, and the added associated data after the format conversion can be written into the associated query table.
[0089] Optionally, after updating the association query table, the updated association query table can be used to query the number of subgraphs, determine whether multiple endpoints belong to the same subgraph, etc.
[0090] For example, when a user needs to determine whether multiple endpoints belong to the same subgraph (for example, the user wants to query whether target endpoints 5, 6, and 10 belong to the same subgraph), the user can enter relevant parameters according to the prompt information in the visualization interface, such as entering the target endpoint ID, and submit it to generate a subgraph attribution query request including the target endpoint ID, where the number of target endpoints is two or more.
[0091] Furthermore, the subgraph attribution query request including the target endpoint id sent by the user can be obtained; according to the target endpoint id, the associated query table is queried; according to the query result, it is determined whether the target endpoints belong to the same subgraph, and the judgment result is fed back to the user. For example, the target endpoints 5, 6 and 10 can be used as input respectively, and the associated query table can be queried by the query function scan to obtain the corresponding row key (i.e., ROWKEY), and then the subgraph id corresponding to each target endpoint can be obtained by parsing the row key; the subgraph id corresponding to each target endpoint obtained is comprehensively analyzed for statistical analysis to determine whether the target endpoints belong to the same subgraph. Furthermore, after obtaining the subgraph id corresponding to each target endpoint, it can be determined whether the target endpoints belong to the same subgraph based on the key-value pair format. For example, the subgraph id corresponding to any target endpoint can be used as the key. If the value includes the ids of each target endpoint, it can be determined that the target endpoints belong to the same subgraph.
[0092] In addition, when the user has a sub-graph number query requirement, he can also input relevant parameters, such as sub-graph data query identifier, etc., according to the prompt information in the visual interface, and submit it to generate a sub-graph number query request. Then, the sub-graph number query request sent by the user can be obtained, the sub-graph id can be extracted from the associated query table, and deduplication can be performed, and then the number of sub-graph ids after deduplication can be counted, and the number of sub-graph ids counted can be used as the number of sub-graphs.
[0093] S360: Use the new version detail data of the target subgraph to update the detail query table including the detail data of the existing subgraph.
[0094] Optionally, in order to query data with low latency, the data in the detail query table in this embodiment can also be stored in a column storage manner, and further can be stored in a column storage manner based on HBase. For example, the data storage structure based on the HBase storage engine is shown in Table 2. Among them, the row key length is 18 bytes, and the row key includes the subgraph id, edge type and edge id. The subgraph id and edge id are both long integers with a length of 8 bytes; the edge type is a numerical value encoded by a dictionary, and the length is 2 bytes. The component adopts a two-level column cluster storage mechanism. There is an attribute column in the column cluster, namely edge information (connection), which is 20 bytes in length; among them, the first endpoint id and the second endpoint id are both long integers with a length of 8 bytes; the first endpoint type and the second endpoint type are both numerical values encoded by a dictionary, and the length is 2 bytes.
[0095]
[0096] Table 2 Detailed data storage structure based on HBase storage engine
[0097] Furthermore, after determining the new details associated data of the target subgraph, the new version details data of the target subgraph can be formatted first, that is, the new version details data of the target subgraph can be converted into query structure data; then, the data after format conversion can be used to update the relevant details query table including the details data of the existing subgraph. Furthermore, the new version details data of the target subgraph can include the modified details data and / or added details data of the target subgraph, and the modified details data after format conversion can be used to replace the relevant data in the details query table, and the added details data after format conversion can be written into the details query table.
[0098] Optionally, after the details query table is updated, the updated details query table can be used to query any sub-graph detail data (ie, sub-graph structure) and the like.
[0099] For example, when a user needs to query detailed data of any subgraph, he or she can enter relevant parameters according to the prompts in the visualization interface, such as entering a single endpoint ID, and submit it to generate a subgraph detail query request including the single endpoint ID.
[0100] Furthermore, the subgraph detail query request including the single endpoint id sent by the user can be obtained; according to the single endpoint id, the detail query table is queried to obtain the detail data of the subgraph associated with the single endpoint; and the detail data of the subgraph associated with the single endpoint is fed back to the user. For example, the single endpoint id can be used as input, and the detail query table can be queried by the query function scan to obtain the corresponding component and row key, and then the id of the subgraph associated with the single endpoint can be obtained by parsing the row key; with the subgraph id as the index, the detail query table can be queried by the query function scan to obtain all components included in the subgraph, and the edge information in the component can be parsed to construct a graph structure object, such as [(edge id, edge type, first endpoint id, first endpoint type, second endpoint id, second endpoint type), ...], and then the graph structure object can be fed back to the user, that is, the detail data of the subgraph associated with the single endpoint can be fed back to the user.
[0101] The technical solution provided by the embodiment of the present invention can quickly query sub-graph related data such as the number of sub-graphs and sub-graph detail data by introducing an associated query table and a detail query table, thereby providing an optional method for quickly querying sub-graph related data.
[0102] Embodiment 4
[0103] Figure 4 Schematic diagram of a graph data processing process provided in Embodiment 4 of the present invention. This embodiment provides a preferred example based on the above embodiment. Figure 4 , the method can be specifically as follows:
[0104] The calculation identifier, point data storage path, edge data storage path and other parameters input by the user are extracted from the visualization interface, and after the calculation identifier is identified as an incremental calculation identifier, the graph data is incrementally calculated. Specifically, the point data and edge data of this batch (i.e., incremental graph data) can be read from the point data storage path and the edge data storage path respectively according to the data storage timestamp, and then the incremental associated data of the target subgraph can be determined according to the point data and edge data in the incremental graph data. The specific process can be referred to the above embodiment, and this embodiment will not be repeated.
[0105] After determining the incremental associated data of the target subgraph, an edge object can be constructed based on the edge data in the newly added data of the target subgraph; and the incremental detailed data of the target subgraph can be determined based on the incremental associated data of the target subgraph and the constructed edge object. n Indicates that the edge object is edgeFrame, eId represents the edge id, eType represents the edge type, fVId represents the first endpoint id, fVType represents the first endpoint type, tVId represents the second endpoint id, and tVType represents the second endpoint type. The structure of the edge object can be expressed as (eId, eType, VId, fVType, tVId, tVType). Then, based on SQL statements, using an open source graph computing framework (such as the Spark GraphX framework), the process of determining the incremental detail data of the target subgraph can be described as: select A n .gId, edgeFrame.eId, edgeFrame.eType, edgeFrame.fVId, edgeFrame.fVType, edgeFrame.tVId, edgeFrame.tVType from A n join edgeFrame on A n .eId=edgeFrame.fVId.
[0106] Further, after determining the incremental associated data of the target subgraph, the associated data of the existing subgraph calculated in the previous batch can be loaded according to the calculation time sequence of the batch. In this process, the endpoint id in the associated data is overwritten (that is, the same endpoint id is read only once), and the overwriting order is the added associated data of the subgraph calculated for the n-1th time> the modified associated data of the subgraph calculated for the n-1th time> the added associated data of the subgraph calculated for the n-2th time...> the modified associated data of the subgraph calculated for the 1st time. And the associated data of the target subgraph is obtained according to the associated data of the existing subgraph, and then the new version associated data of the target subgraph can be determined according to the incremental associated data and the original associated data of the target subgraph. The new version associated data of the target subgraph can include the modified associated data of the target subgraph and / or the added associated data of the target subgraph. Or the new version associated data of the target subgraph can be determined directly according to the associated data of the existing subgraph and the incremental associated data of the target subgraph. For example, the associated data of the existing subgraph can be expressed as ∑A, the subgraph id can be expressed as gId, and the endpoint id can be expressed as vId. Then, based on SQL statements, using an open source graph computing framework (such as the Spark GraphX framework), the process of modifying the associated data in the new version of the target subgraph can be described as: select A n .vId,A n .gId,(∑A).gIdas oGId from A n join(∑A)on A n .vId=(∑A).vId;Determine the new version of the target subgraph. The process of adding associated data can be described as: select A n .vId,A n .gId from A n where A n .vId except(∑A).vId.
[0107] Optionally, after obtaining the new version of the associated data of the target subgraph, the new version of the associated data of the target subgraph can be stored in the file system. Further, to facilitate subsequent reading and based on the limitation of the running memory, the added associated data and the modified associated data in the new version of the associated data of the target subgraph in this embodiment can be stored in different files.
[0108] Further, after obtaining the new version associated data of the target subgraph, the added detail data in the new version detailed data of the target subgraph can be determined according to the added associated data of the target subgraph and the edge data in the newly added data of the target subgraph. The specific process can be referred to in the above embodiment, which will not be described in detail in this embodiment. For example, the added associated data of the target subgraph is represented by A n(Newly added), based on SQL statements, using an open source graph computing framework (such as the Spark GraphX framework), the process of determining the added detail data in the new version detail data of the target subgraph can be described as: select A n (Newly added).gId, edgeFrame.eId, edgeFrame.eType, edgeFrame.fVId, edgeFrame.fVType, edgeFrame.tVId, edgeFrame.tVType from A n (New)join edgeFrame on A n (New).vId=min(edgeFrame.fVId, edgeFrame.tVId).
[0109] Similarly, read the edge data of the previous batch, remove duplicates according to the edge number, and obtain the edge data in the original data of the target subgraph from the edge data after deduplication. Then, the edge object can be constructed based on the edge data in the original data of the target subgraph; according to the modified associated data of the target subgraph and the constructed edge object, the modified detail data in the new version of the target subgraph is determined. Alternatively, the edge object, i.e., ∑edgeFrame, can be directly constructed based on the edge data after deduplication, and then the modified detail data in the new version of the target subgraph is determined based on the modified associated data of the target subgraph and the constructed edge object. For example, the modified associated data of the target subgraph is represented as A n (Modify), then based on SQL statements, using an open source graph computing framework (such as the Spark GraphX framework), the process of determining the modified details in the new version details of the target subgraph can be described as: select A n (modification).gId, (∑edgeFrame).eId, (∑edgeFrame).eType, (∑edgeFrame).fVId, (∑edgeFrame).fVType, (∑edgeFrame).tVId, (∑edgeFrame).tVType from A n (Modify)join(∑edgeFrame)on A n (modification).vId=min((∑edgeFrame).fVId, (∑edgeFrame).tVId).
[0110] Optionally, after obtaining the new version detail data of the target subgraph, the new version detail data of the target subgraph may also be stored in a file system. Further, to facilitate subsequent reading and based on the limitation of the running memory, the added detail data and the modified detail data in the new version detail data of the target subgraph in this embodiment may be stored in different files.
[0111] Afterwards, the operation of format conversion of the new version detail data of the target subgraph and the new version associated data of the target subgraph can be performed, as well as the operation of updating the associated query table and the detail query table; then, based on the updated associated query table, the number of subgraphs can be queried, whether multiple endpoints belong to the same subgraph, etc. can be determined, and based on the updated detail query table, any subgraph detail data (i.e., subgraph structure) can be queried, etc. The specific process can be referred to the above embodiment, which will not be repeated here. It should be noted that the data after format conversion is stored in the temporary space of the file system, so the data after format conversion can be transferred from the temporary space to the persistent space, such as the hard disk.
[0112] Exemplarily, if the calculation identifier is identified as a full calculation identifier, the graph data is fully calculated. Optionally, in the process of performing the full calculation on the graph data, the process is similar to the process of processing the incremental graph data in the above embodiment to obtain the incremental associated data of the target subgraph. The specific processing process is as follows: all batches of point data (i.e., total point data) can be read from the point data storage path, and a first total point set can be extracted from the read total point data; all batches of edge data (i.e., total edge data) can be read from the edge data storage path, and the endpoints associated with the edges can be extracted from the total edge data, the extracted endpoints are deduplicated, and the deduplicated endpoints are used as the second total point set. Afterwards, the difference between the first total point set and the second total point set can be used as an isolated total point set; and the isolated total point set can be used as the first total subgraph.
[0113] At the same time, a graph object can be constructed based on the total edge data, and an open source graph computing framework (such as the Spark GraphX framework) can be used to perform subgraph computing to obtain a second total subgraph. Figure 1 And it serves as the target total subgraph; and based on the storage method of the associated data, the associated data of the target total subgraph can be obtained according to the point data in the target total subgraph.
[0114] In addition, after obtaining the association data of the target total subgraph, the total edge object can be obtained according to the total edge data, and then the detail data of the target total subgraph can be obtained according to the association data and the total edge object of the target total subgraph. Optionally, in this embodiment, the association data of the target total subgraph and the detail data of the target total subgraph can also be stored in the file system.
[0115] Afterwards, the operation of formatting the detailed data of the target total subgraph and the associated data of the target total subgraph can be performed, and the formatted data can be loaded into the associated query table and the detailed query table respectively. Then, based on the associated query table, the number of subgraphs can be queried, and whether multiple endpoints belong to the same subgraph can be determined. Based on the detailed query table, the detailed data of any subgraph (i.e., the subgraph structure) can be queried, etc. The specific process can be referred to the above embodiment, which will not be repeated here.
[0116] It should be noted that, in this embodiment, based on providing a visual interface, the user can flexibly select a calculation method, which further increases the flexibility of the solution.
[0117] The technical solution provided by the embodiment of the present invention lays a foundation for fast subgraph calculations, etc. by introducing the associated data and detailed data of the subgraph; at the same time, the incremental graph data is processed to obtain the new version associated data and new version detailed data of the target subgraph for subgraph calculations. Compared with the existing offline calculation method, this embodiment only processes the incremental graph data, which reduces the amount of data calculation, improves the data processing efficiency, and can meet the user's needs for subgraph calculations. In addition, by introducing the associated query table and the detailed query table, subgraph-related data such as the number of subgraphs and subgraph detailed data can be quickly queried, providing an optional method for quickly querying subgraph-related data.
[0118] Embodiment 5
[0119] Figure 5 This is a structural block diagram of a graph data processing device provided in Embodiment 5 of the present invention. The device can execute the graph data processing method provided in any embodiment of the present invention and has the corresponding functional modules and beneficial effects of the execution method. Figure 5 As shown, the device may include:
[0120] A target data determination module 510 is used to determine a target subgraph and newly added data of the target subgraph according to the incremental graph data;
[0121] The incremental associated data determination module 520 is used to determine the incremental associated data of the target subgraph according to the midpoint data of the newly added data of the target subgraph;
[0122] A new associated data determination module 530 is used to determine the new version associated data of the target subgraph according to the incremental associated data and the original associated data of the target subgraph;
[0123] A new detail data determination module 540, configured to determine the new version detail data of the target subgraph according to the new version association data of the target subgraph and the edge data in the target subgraph;
[0124] The associated data includes the endpoint id and subgraph id; the detailed data includes the subgraph id, edge id, edge type, edge-associated endpoint id and endpoint type.
[0125] The technical solution provided by the embodiment of the present invention lays the foundation for fast subgraph calculation by introducing the associated data and detailed data of the subgraph; at the same time, the incremental graph data is processed to obtain the new version associated data and new version detailed data of the target subgraph for subgraph calculation. Compared with the existing offline calculation method, this embodiment only processes the incremental graph data, which reduces the amount of data calculation, improves the data processing efficiency, and can meet the user's needs for subgraph calculation.
[0126] Exemplarily, the incremental associated data determination module 520 may be specifically used for:
[0127] Determine the first id of the target subgraph according to the endpoint id in the newly added data of the target subgraph;
[0128] The incremental associated data of the target subgraph is determined according to the first id of the target subgraph and the endpoint id in the newly added data of the target subgraph.
[0129] Exemplarily, the new associated data determination module 530 may be specifically used for:
[0130] Determine the target id of the target subgraph according to the first id of the target subgraph in the incremental association data and the second id of the target subgraph in the original association data;
[0131] According to the target id of the target subgraph and the endpoint id in the target subgraph, new version associated data of the target subgraph is determined; the new version associated data of the target subgraph includes modified associated data of the target subgraph and / or added associated data of the target subgraph.
[0132] Exemplarily, the new details data determination module 540 may be specifically used for:
[0133] Determine the modified detail data in the new version detail data of the target subgraph according to the modified association data of the target subgraph and the edge data in the original data of the target subgraph;
[0134] According to the added associated data of the target subgraph and the edge data in the newly added data of the target subgraph, the added detail data in the new version detail data of the target subgraph is determined.
[0135] Exemplarily, the above device may further include: an update module, which may be specifically used to:
[0136] Using the new version of the target subgraph's associated data, the associated query table including the associated data of the existing subgraph is updated;
[0137] Using the new version of the target subgraph's detail data, the detail query table including the detail data of the existing subgraph is updated;
[0138] The data in the associated query table and the detail query table are stored in columnar storage.
[0139] Exemplarily, the above device may further include: a same sub-image determination module, which may be specifically used for:
[0140] Obtaining a subgraph attribution query request including a target endpoint ID sent by a user; the number of target endpoints is at least two;
[0141] According to the target endpoint id, query the associated query table;
[0142] Based on the query results, determine whether the target endpoints belong to the same subgraph, and feedback the judgment result to the user.
[0143] Exemplarily, the above device may further include: a sub-graph detail query module, which may be specifically used for:
[0144] Get the subgraph details query request including the single endpoint id sent by the user;
[0145] According to the single endpoint ID, query the detail query table to obtain the detail data of the subgraph associated with the single endpoint;
[0146] Feedback to the user the detailed data of the subgraph associated with a single endpoint.
[0147] Embodiment 6
[0148] Figure 6 This is a schematic diagram of the structure of an electronic device provided in Embodiment 6 of the present invention. Figure 6 A block diagram of an exemplary electronic device 12 suitable for use in implementing embodiments of the present invention is shown. Figure 6 The electronic device 12 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention. Optionally, the electronic device can typically be a server that carries the image data processing function.
[0149] like Figure 6 As shown, the electronic device 12 is in the form of a general purpose computing device. The components of the electronic device 12 may include, but are not limited to: one or more processors or processing units 16, a memory 28, and a bus 18 that connects various system components (including the memory 28 and the processing unit 16).
[0150] Bus 18 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor or a local bus using any of a variety of bus architectures. By way of example, these architectures include, but are not limited to, an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MAC) bus, an Enhanced ISA bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus.
[0151] The electronic device 12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the electronic device 12, including volatile and non-volatile media, removable and non-removable media.
[0152] The memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. The electronic device 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 34 may be used to read and write non-removable, non-volatile magnetic media ( Figure 6 not shown, usually called a "hard drive"). Although Figure 6 Not shown in the figure, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk"), and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, a DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to the bus 18 via one or more data medium interfaces. The memory 28 may include at least one program product having a set (e.g., at least one) of program modules that are configured to perform the functions of the various embodiments of the present invention.
[0153] A program / utility 40 having a set (at least one) of program modules 42 may be stored, for example, in the memory 28, such program modules 42 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment. The program modules 42 generally perform the functions and / or methods of the embodiments described herein.
[0154] The electronic device 12 may also communicate with one or more external devices 14 (e.g., keyboards, pointing devices, displays 24, etc.), may communicate with one or more devices that enable a user to interact with the electronic device 12, and / or may communicate with any device that enables the electronic device 12 to communicate with one or more other computing devices (e.g., a network card, a modem, etc.). Such communication may be performed via an input / output (I / O) interface 22. Furthermore, the electronic device 12 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 20. As shown, the network adapter 20 communicates with other modules of the electronic device 12 via the bus 18. It should be understood that although Figure 6 Not shown, other hardware and / or software modules may be used in conjunction with the electronic device 12, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.
[0155] The processing unit 16 executes various functional applications and data processing by running the programs stored in the memory 28, such as implementing the graph data processing method provided in the embodiment of the present invention.
[0156] Embodiment 7
[0157] Embodiment 7 of the present invention further provides a computer-readable storage medium on which a computer program (or computer-executable instructions) is stored. When the program is executed by a processor, it is used to execute the graph data processing method provided in the embodiment of the present invention.
[0158] The computer storage medium of the embodiment of the present invention can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, a device or a device or used in combination with it.
[0159] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.
[0160] The program code embodied on the computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
[0161] Computer program code for performing the operations of the present invention may be written in one or more programming languages or a combination thereof, including object-oriented programming languages such as Java, Smalltalk, C++, and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).
[0162] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.
Claims
1. A graph data processing method, characterized in that: include: Determine the target subgraph and the newly added data of the target subgraph according to the incremental graph data; Determining the incremental associated data of the target subgraph according to the midpoint data of the newly added data of the target subgraph; Determining new version associated data of the target subgraph according to the incremental associated data and the original associated data of the target subgraph; Determine new version detail data of the target subgraph according to the new version association data of the target subgraph and the edge data in the target subgraph; The associated data includes the endpoint id and subgraph id; the detailed data includes the subgraph id, edge id, edge type, the endpoint id and endpoint type associated with the edge; Determining the incremental associated data of the target subgraph according to the midpoint data of the newly added data of the target subgraph includes: Determine the first id of the target subgraph according to the endpoint id in the newly added data of the target subgraph; Determine the incremental associated data of the target subgraph according to the first id of the target subgraph and the endpoint id in the newly added data of the target subgraph; Determining new version associated data of the target subgraph according to the incremental associated data and the original associated data of the target subgraph includes: Determine a target id of the target subgraph according to a first id of the target subgraph in the incremental association data and a second id of the target subgraph in the original association data; Determine the new version associated data of the target subgraph according to the target id of the target subgraph and the endpoint id in the target subgraph; the new version associated data of the target subgraph includes the modified associated data of the target subgraph and / or the added associated data of the target subgraph; The subgraph id is the smallest or largest endpoint id among the endpoints included in the subgraph; After determining the new version associated data of the target subgraph, the method further includes: Using the new version of the target subgraph's associated data, updating the associated query table including the associated data of the existing subgraph; Using the new version detail data of the target subgraph, updating the detail query table including the detail data of the existing subgraph; The data in the association query table and the details query table are stored in a column-based storage manner; Obtaining a subgraph attribution query request including a target endpoint ID sent by a user; the number of target endpoints is at least two; According to the target endpoint ID, query the association query table; According to the query result, determine whether the target endpoints belong to the same subgraph, and feedback the determination result to the user; After determining the incremental associated data of the target subgraph, the incremental associated data of the target subgraph is saved to a temporary space of the file system according to the calculation timestamp; Determining new version detail data of the target subgraph according to the new version association data of the target subgraph and the edge data in the target subgraph includes: Determining modified detail data in new version detail data of the target subgraph according to the modified associated data of the target subgraph and edge data in original data of the target subgraph; Determine, according to the added association data of the target subgraph and the edge data in the newly added data of the target subgraph, the added detail data in the new version detail data of the target subgraph; Also includes: Get the subgraph details query request including the single endpoint id sent by the user; According to the single endpoint ID, query the detail query table to obtain the detail data of the subgraph associated with the single endpoint; Feedback the user with detailed data of the subgraph associated with the single endpoint.
2. A graph data processing device, used to execute the graph data processing method according to claim 1, characterized in that: include: A target data determination module, used to determine a target subgraph and newly added data of the target subgraph according to the incremental graph data; An incremental associated data determination module, used to determine the incremental associated data of the target subgraph according to the midpoint data of the newly added data of the target subgraph; A new associated data determination module, used to determine the new version associated data of the target subgraph according to the incremental associated data and the original associated data of the target subgraph; A new detail data determination module, used to determine the new version detail data of the target subgraph according to the new version association data of the target subgraph and the edge data in the target subgraph; The associated data includes the endpoint id and subgraph id; the detailed data includes the subgraph id, edge id, edge type, the endpoint id and endpoint type associated with the edge; Determining the incremental associated data of the target subgraph according to the midpoint data of the newly added data of the target subgraph includes: Determine the first id of the target subgraph according to the endpoint id in the newly added data of the target subgraph; Determine the incremental associated data of the target subgraph according to the first id of the target subgraph and the endpoint id in the newly added data of the target subgraph; Determining new version associated data of the target subgraph according to the incremental associated data and the original associated data of the target subgraph includes: Determine a target id of the target subgraph according to a first id of the target subgraph in the incremental association data and a second id of the target subgraph in the original association data; Determine the new version associated data of the target subgraph according to the target id of the target subgraph and the endpoint id in the target subgraph; the new version associated data of the target subgraph includes the modified associated data of the target subgraph and / or the added associated data of the target subgraph; The subgraph id is the smallest or largest endpoint id among the endpoints included in the subgraph; After determining the new version associated data of the target subgraph, the method further includes: Using the new version of the target subgraph's associated data, updating the associated query table including the associated data of the existing subgraph; Using the new version detail data of the target subgraph, updating the detail query table including the detail data of the existing subgraph; The data in the association query table and the details query table are stored in a column-based storage manner; Obtaining a subgraph attribution query request including a target endpoint ID sent by a user; the number of target endpoints is at least two; According to the target endpoint ID, query the association query table; According to the query result, determine whether the target endpoints belong to the same subgraph, and feedback the determination result to the user; After determining the incremental associated data of the target subgraph, the incremental associated data of the target subgraph is saved to a temporary space of the file system according to the calculation timestamp; The device also includes a subgraph detail query module, which is used to obtain a subgraph detail query request including a single endpoint ID sent by a user; query the detail query table according to the single endpoint ID to obtain detail data of the subgraph associated with the single endpoint; and feed back the detail data of the subgraph associated with the single endpoint to the user; The new detail data determination module is also used to: determine the modified detail data in the new version detail data of the target subgraph according to the modified associated data of the target subgraph and the edge data in the original data of the target subgraph; determine the added detail data in the new version detail data of the target subgraph according to the added associated data of the target subgraph and the edge data in the newly added data of the target subgraph.
3. An electronic device, characterized in that: include: one or more processors; A storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the graph data processing method as described in claim 1.
4. A medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the graph data processing method as described in claim 1 is implemented.
