Graph database construction method and system, medium and equipment

Abstract

The invention discloses a graph database construction method and system, a medium and equipment, and relates to the technical field of data query. The method comprises the following steps: preparing data according to a data query requirement; inserting a chart according to the prepared data and setting chart parameters to obtain an initial chart database; analyzing attributes of the graph database by utilizing a graph search algorithm and a query statement, and traversing to obtain all elements and paths; filtering, gathering and optimizing the paths by utilizing the query statement to obtain a transition graph database; the transition graph database is adjusted and optimized, a final graph database is obtained, and the adjustment and optimization step comprises the steps of conducting traversal railing optimization on the transition graph database, conducting statistics on the transition graph database and conducting structure adjustment and optimization on the graph database. The graph database constructed by the method supports various graph traversal algorithms and pattern matching, and large-scale graph data can be efficiently queried and analyzed.

Description

Technical Field

[0001] The present invention relates to the field of data query technology, and in particular to a method, system, medium and device for constructing a graph database. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] When the concept of graph databases was not yet common and there was no dedicated query language, Java API was mainly used for graph query. Later, Neo4j released the first version of Cypher, which became a declarative language. Then, graph query languages ​​developed rapidly, including Oracle's PGQL and Neo4j's Cypher. Currently, GQL has become an international standard, aiming to unify the standards of different graph query languages.

[0004] However, Neo4j hopes to construct data in the form of a network, that is, a graph. At this time, the concept of graph database is not clear, and there is no query language, so it is done using Java API, using 2-3 interfaces to obtain points, and then based on a point, obtain its first-degree initial edge, and describe a graph query in this form. Now it seems that whether it is JAVA API or Neo4j, the process of graph query follows the original idea of ​​​​graph query, just like TuGraph originally had an API layer first, and then built a query language on the upper layer. However, the above query method does not support multiple data models, nor does it include graphs, lists, trees, etc. It is also not easy to optimize research and development, the query process is complicated, and it does not support the standard GQL language. Summary of the invention

[0005] In view of the shortcomings of the prior art, the purpose of the present invention is to provide a method, system, medium and device for constructing a graph database. The graph database constructed by the present invention supports multiple graph traversal algorithms and pattern matching, and can efficiently query and analyze large-scale graph data.

[0006] In order to achieve the above object, the present invention is implemented through the following technical solutions:

[0007] The first aspect of the present invention provides a construction method based on a graph database, comprising the following steps:

[0008] Prepare data according to data query requirements;

[0009] Insert the chart according to the prepared data and set the chart parameters to obtain the initial chart database;

[0010] Use graph search algorithms and query statements to analyze the properties of the graph database and traverse to obtain all elements and paths;

[0011] Filter, aggregate and optimize the paths using query statements to obtain a transition graph database;

[0012] The transition graph database is adjusted and optimized to obtain a final graph database, wherein the adjustment and optimization steps include traversal railing optimization of the transition graph database, statistics of the transition graph database, and structural adjustment and optimization of the graph database.

[0013] Furthermore, the specific steps to insert a chart and set chart parameters according to the prepared data are as follows:

[0014] Select the chart type based on the prepared data;

[0015] Insert a chart according to the chart type;

[0016] Format and style your chart.

[0017] Furthermore, the specific steps of analyzing the attributes of the graph database using the graph search algorithm and query statements are as follows:

[0018] Traverse the graph data set according to vertex distribution;

[0019] During the traversal process, the dimensions to be grouped are specified through attributes of the same type;

[0020] After visiting each adjacent vertex in turn, start from these adjacent vertices and visit the adjacent vertices that have not been visited in turn;

[0021] Use a first-in-first-out queue to save the vertices that have been visited, so that the adjacent points of the first-visited vertices are visited first in the next round of visits;

[0022] The visiting process is repeated until all vertices in the graph that have the same path as the starting point have been visited.

[0023] Furthermore, the specific steps of using query statements to filter, aggregate and optimize paths are as follows:

[0024] Traverse each node in the path and select the path that meets the query statement in the path results;

[0025] When traversing each node in the path, the fields of the specified attributes are used to modify the single-step query to achieve field aggregation;

[0026] Label the aggregated fields before merging and copying them.

[0027] Furthermore, the specific steps of optimizing the traversal railing of the transition graph database are as follows:

[0028] Insert the fence into the lazy loading traversal pipeline, and ensure that the steps before the fence operation are completed before continuing to execute;

[0029] Use hugegraph fences to force a change from depth-first search to breadth-first search.

[0030] Optimize large amounts of repeated data access through layer bulking mode.

[0031] Furthermore, the specific steps for statistical analysis of the transition graph database are as follows:

[0032] Use the explain() function to parse how the description statement is transformed into the final set of steps to be executed during compilation;

[0033] Use profile() to count the time consumed by each step in the execution of the statement and the objects passed.

[0034] Furthermore, the specific steps for structural adjustment and optimization of the graph database are as follows:

[0035] Modify the data storage structure of the graph database;

[0036] Set up and select the appropriate index;

[0037] Partition the data based on its characteristics;

[0038] Add caching solution.

[0039] A second aspect of the present invention provides a system for constructing a graph database, comprising:

[0040] A graph database creation module is configured to prepare data according to data query requirements; insert a graph according to the prepared data and set graph parameters to obtain an initial graph database;

[0041] The graph database arrangement module is configured to analyze the attributes of the graph database using a graph search algorithm and a query statement, traverse to obtain all elements and paths; use the query statement to filter, aggregate and optimize the paths to obtain a transition graph database;

[0042] The graph database optimization module is configured to adjust and optimize the transition graph database to obtain the final graph database, wherein the adjustment and optimization steps include traversal railing optimization of the transition graph database, statistics of the transition graph database, and structural adjustment and optimization of the graph database.

[0043] The third aspect of the present invention provides a medium on which a program is stored, and when the program is executed by a processor, the steps in the method for constructing a graph database as described in the first aspect of the present invention are implemented.

[0044] The fourth aspect of the present invention provides a device, including a memory, a processor, and a program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps in the method for constructing a graph database as described in the first aspect of the present invention are implemented.

[0045] One or more of the above technical solutions have the following beneficial effects:

[0046] The present invention discloses a method, system, medium and device for constructing a graph database. The graph database constructed by the present invention has the advantages of flexibility, powerful graph traversal capability, scalability and concurrency. The graph data constructed by the present invention has high flexibility, and complex query logic can be written according to specific needs, supporting multiple graph traversal operations, such as vertex filtering, edge filtering, path traversal, etc. In addition, a rich set of traversal steps and operators are provided, which can be combined and customized according to specific needs, supporting complex query logic and conditional filtering. The graph data constructed by the present invention supports multiple graph traversal algorithms and pattern matching, and can efficiently query and analyze large-scale graph data. Allow users to perform complex query operations by traversing the vertices and edges in the graph, and perform depth-first or breadth-first traversal in the graph to discover and operate related data in the graph.

[0047] The graph data constructed by the present invention can be easily expanded and customized, and customized traversal steps and functions can be added according to actual needs. In addition, it can be integrated and upgraded with various graph databases, including JanusGraph, Neo4j, etc., and can also be combined with other computing frameworks (such as Spark, Hadoop) to achieve large-scale graph computing and distributed query. The graph data constructed by the present invention can be executed concurrently, making full use of multi-core processors and distributed computing resources to improve query efficiency and throughput. It enables higher performance and faster response time when processing large-scale data. The optimized graph database of the present invention can effectively improve query efficiency and enhance the scalability of the system.

[0048] Advantages of additional aspects of the present invention will be given in part in the following description, and in part will become obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0050] Figure 1 This is a flow chart of a method for constructing a graph database in Embodiment 1 of the present invention;

[0051] Figure 2This is a flow chart of the path filtering steps in the first embodiment of the present invention. DETAILED DESCRIPTION

[0052] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0053] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "include" and / or "include" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or their combinations;

[0054] Embodiment 1:

[0055] The first embodiment of the present invention provides a construction method based on a graph database. The steps of creating a graph mainly include preparing data, selecting a graph type, inserting a graph, setting a graph format, and adjusting a graph style. Figure 1 As shown, it includes metadata modeling, data import, and data analysis.

[0056] The specific steps include:

[0057] Step 1: Prepare data according to data query requirements.

[0058] In a specific implementation, preparing data is the basis for creating a chart. Enter the data table to be charted in Excel or a similar form, usually a general table without details, and then select the data table.

[0059] Step 2: Insert the chart according to the prepared data and set the chart parameters to obtain the initial graph database.

[0060] Step 2.1: Select the chart type based on the prepared data.

[0061] In a specific implementation, click "Insert-Chart-Recommended Chart" on the menu bar in Excel, then select the chart type on the left and the chart you want to create on the right. For example, you can select a clustered column chart, a line chart, a pie chart, etc.

[0062] Step 2.2: Insert a chart according to the chart type.

[0063] In a specific implementation, a chart is created based on the selected data. Excel will preset many chart styles, which can be directly selected and used, or the style, color, data label, etc. can be customized according to needs.

[0064] Step 2.3: Format and style the chart.

[0065] In a specific implementation, the title, legend, coordinate axes, etc. of the chart can be adjusted to ensure that the chart is clear and easy to understand. The appearance and content of the chart can be modified through the "Chart Design" and "Format" tabs.

[0066] Step 3: Use graph search algorithms and query statements to analyze the attributes of the graph database, perform grouping and deduplication, and traverse to obtain all elements and paths.

[0067] Step 3.1: Traverse the graph data set according to vertex distribution. Starting from a vertex v, traverse using the breadth-first algorithm.

[0068] Step 3.2: During the traversal process, specify the dimension to be grouped by using attributes of the same type.

[0069] In a specific implementation, a map set and a linkList set are first set. During the traversal process, the dimensions to be grouped are specified by using attributes of the same type. Elements of the same dimension are placed in the map, and the attribute value is used as the key of the map. The linkList set is then used to store the map value to remove duplication.

[0070] Step 3.3: After visiting each adjacent vertex in turn, start from these adjacent vertices and visit the adjacent vertices that have not been visited in turn.

[0071] Step 3.4: Use a first-in-first-out queue to save the vertices that have been visited, so that the adjacent points of the first-visited vertices are visited first in the next round of visits.

[0072] In a specific implementation, during the traversal process, each time a vertex is visited, it is enqueued, and when the head element is dequeued, its unvisited adjacent vertices are enqueued, and each vertex is enqueued only once.

[0073] Step 3.5: Repeat the visiting process until all vertices in the graph that have the same path as the starting point have been visited.

[0074] Step 3.6: Count and get the elements and paths of each set in the map.

[0075] Step 4: Use query statements to filter, aggregate and optimize the paths to obtain a transition graph database.

[0076] Step 4.1: Filtering process: traverse each node in the path and select the path that meets the query statement in the path results.

[0077] In a specific embodiment, Figure 2 As shown in the figure, the obtained path results are selected. First, each step on the way is labeled by the out() method, and then the aggregate() method is used to loop through the results of several steps as new results and merge them. In addition, the in() method is used to specify the dimension to convert the input. Then the path results are selected by matching the where function conditions, so that the paths that meet the conditions are selected.

[0078] Step 4.2: Aggregation process: When traversing each node in the path, use the fields of the specified attributes to modify the single-step query and implement field aggregation.

[0079] In a specific implementation, during the traversal of each node in the path, the input in is aggregated into a list <object>The list generated by the single step is saved in the path. The single step will not affect the input objects, and the next single step will still use objects as input. <object>It is read-only, and the value can be obtained through methods at any position of subsequent traversal.

[0080] A single step can be modified by a field of a specified attribute to specify the aggregated field value. Specifically, after specifying the attribute field modifier, the fields of the same attribute can be classified and merged to achieve aggregation and summary, which is convenient for subsequent search queries.

[0081] Single-step can implicitly insert delimiters, and using storage can implement lazy loading aggregation without blocking. Specifically, the operation in the Greml in query syntax can process the input object during the traversal process and insert delimiters when necessary. This process implicitly inserts delimiters to separate different vertex names, which can be used to implement lazy loading aggregation.

[0082] This example does not recommend using the selector select() function to retrieve values. Since the selector function returns a list for each entity, <object>, multiple entities will return multiple identical lists <object>,This not only increases the complexity of the query steps, but also increases the time dimension of program processing.

[0083] Step 4.3: Optimization process: Label the aggregated fields, merge and copy them, so that the fields can be quickly filtered according to the label values.

[0084] In a specific implementation, the merge is to aggregate the input into list and copy it to the side information. The next step takes objects as input, and the copied value is read-only for subsequent traversals. The as function is to label the input elements. The next step takes the labeled object set as input. The labeled object set is modifiable in the subsequent traversal algorithm.

[0085] Step 5: Adjust and optimize the transitional graph database to obtain the final graph database.

[0086] Among them, the adjustment and optimization steps include traversal railing optimization of the transition graph database, statistics of the transition graph database, and structural adjustment and optimization of the graph database.

[0087] Step 5.1: Optimize the traversal rails on the transition graph database.

[0088] Step 5.1.1: Insert the fence into the lazy loading traversal pipeline, and ensure that the steps before the fence operation are completed before continuing to execute.

[0089] Step 5.1.2: Use hugegraph fences to force the change from depth-first search to breadth-first search.

[0090] When hugegraph's gremlin language is used to traverse the graph database, the barrier can be inserted into the lazy loaded traversal pipeline to ensure that all steps before the barrier are completed before continuing to execute.

[0091] Step 5.1.3: Optimize large amounts of repeated data access through the layer's bulking mode to improve efficiency.

[0092] Step 5.2: Calculate statistics on the transition graph database.

[0093] Step 5.2.1: Use the explain() function to parse how the description statement is transformed into the final set of steps to be executed during compilation.

[0094] Step 5.2.2: Use profile() to collect statistical information such as the time consumed by each step and the objects passed during the execution of the statistics statement.

[0095] Step 5.3: Adjust and optimize the structure of the graph database.

[0096] Step 5.3.1: Modify the data storage structure of the graph database.

[0097] In a specific implementation, the data storage structure of the graph database is modified, and the nodes and edges of the data storage need to avoid complex data structures and too deep nesting levels, so as to effectively reduce the query time.

[0098] Step 5.3.2: Set up and select the appropriate index.

[0099] In a specific implementation, it is very important to set and select the appropriate index. This is because the graph database model and query method are quite different from other relational databases. You need to choose according to the specific query requirements and condition scenarios. For example, when querying node attributes, select the attribute index; when querying edge attributes, select the edge attribute index; when querying which relationships a node has, select the relationship index. In addition, for combined queries, use composite indexes to improve query efficiency. The specific method is to apply multiple indexes to multiple attributes, which will not be repeated here.

[0100] Step 5.3.3: Partition the data based on its characteristics.

[0101] In a specific implementation, data partitioning is performed, and horizontal partitioning or vertical partitioning can be selected according to the characteristics of the data. Horizontal partitioning divides the data into multiple partitions, each partition stores different data, so as to reduce the storage and access load of a single node. Vertical partitioning can distribute the data of a node in multiple data tables, each table containing different attributes and relationships. The two partitioning methods can be flexibly selected and used according to actual conditions.

[0102] Step 5.3.4: Add caching solution.

[0103] In a specific implementation, caching technology is added. Traditional relational databases support various caching solutions, but graph databases are more difficult to implement due to their complex data models. A very large storage space is required to cache data. Common methods include memory cache and distributed cache. Memory cache uses the read and write capabilities and efficiency of memory to improve query efficiency, while distributed cache is horizontally scalable and highly flexible. You need to consider the size of the data and the number of daily visits, and make your own choices.

[0104] The present invention makes full use of the relevant characteristics of the query algorithm, combined with the characteristics of the graph database itself, to achieve the query of graph attributes and characteristics without introducing other plug-ins, so that rendering loading is more efficient and the code implementation is simple. The present invention provides good support for the query of graph databases by using index query process analysis for small and medium-sized systems without introducing other third-party components. While ensuring efficient query, it can effectively reduce the waste of memory resources, reduce logical processing, and R&D costs. Users can use custom query encapsulation methods to facilitate integration and use, and it contains rich application interfaces, scenarios and parameter configurations, such as aggregation and expansion of results, selection and filtering of paths, grouping and deduplication of data, traversal of fences, etc. The present invention adopts Secondary to support secondary indexes with exact matching, allowing the establishment of joint indexes. The joint index supports indexing, which can effectively solve the problem of slow search in scenarios with extremely large amounts of graph attribute and path data.

[0105] Embodiment 2:

[0106] Embodiment 2 of the present invention provides a system for constructing a graph database, including:

[0107] A graph database creation module is configured to prepare data according to data query requirements; insert a graph according to the prepared data and set graph parameters to obtain an initial graph database;

[0108] The graph database arrangement module is configured to analyze the attributes of the graph database using a graph search algorithm and a query statement, traverse to obtain all elements and paths; use the query statement to filter, aggregate and optimize the paths to obtain a transition graph database;

[0109] The graph database optimization module is configured to adjust and optimize the transition graph database to obtain the final graph database, wherein the adjustment and optimization steps include traversal railing optimization of the transition graph database, statistics of the transition graph database, and structural adjustment and optimization of the graph database.

[0110] Embodiment three:

[0111] Embodiment 3 of the present invention provides a medium on which a program is stored. When the program is executed by a processor, the steps in the construction method based on the graph database as described in Embodiment 1 of the present invention are implemented.

[0112] Embodiment 4:

[0113] Embodiment 4 of the present invention provides a device, including a memory, a processor, and a program stored in the memory and executable on the processor. When the processor executes the program, the steps in the construction method based on the graph database as described in Embodiment 1 of the present invention are implemented.

[0114] The steps involved in the above embodiments 2, 3 and 4 correspond to the method embodiment 1. For the specific implementation methods, please refer to the relevant description part of embodiment 1.

[0115] Those skilled in the art should understand that the modules or steps of the present invention described above can be implemented by a general-purpose computer device, or alternatively, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. The present invention is not limited to any specific combination of hardware and software.

[0116] Although the above describes the specific implementation mode of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without creative work are still within the scope of protection of the present invention.< / object> < / object> < / object> < / object>

Claims

1. A method for constructing a graph database, characterized in that: The following steps are involved: Prepare data according to data query requirements; Insert the chart according to the prepared data and set the chart parameters to obtain the initial chart database; Use graph search algorithms and query statements to analyze the properties of the graph database and traverse to obtain all elements and paths; Filter, aggregate and optimize the paths using query statements to obtain a transition graph database; The transition graph database is adjusted and optimized to obtain a final graph database, wherein the adjustment and optimization steps include traversal railing optimization of the transition graph database, statistics of the transition graph database, and structural adjustment and optimization of the graph database.

2. The method for constructing a graph database according to claim 1, wherein: The specific steps to insert a chart and set chart parameters according to the prepared data are: Select the chart type based on the prepared data; Insert a chart according to the chart type; Format and style your chart.

3. The method for constructing a graph database according to claim 1, characterized in that: The specific steps to analyze the attributes of a graph database using graph search algorithms and query statements are as follows: Traverse the graph data set according to vertex distribution; During the traversal process, the dimensions to be grouped are specified through attributes of the same type; After visiting each adjacent vertex in turn, start from these adjacent vertices and visit the adjacent vertices that have not been visited in turn; Use a first-in-first-out queue to save the vertices that have been visited, so that the adjacent points of the first-visited vertices are visited first in the next round of visits; The visiting process is repeated until all vertices in the graph that have the same path as the starting point have been visited.

4. The method for constructing a graph database according to claim 1, wherein: The specific steps to filter, aggregate and optimize paths using query statements are as follows: Traverse each node in the path and select the path that meets the query statement in the path results; When traversing each node in the path, the fields of the specified attributes are used to modify the single-step query to achieve field aggregation; Label the aggregated fields before merging and copying them.

5. The method for constructing a graph database according to claim 1, wherein: The specific steps for optimizing the traversal railing of the transition graph database are as follows: Insert the fence into the lazy loading traversal pipeline, and ensure that the steps before the fence operation are completed before continuing to execute; Use hugegraph fences to force a change from depth-first search to breadth-first search. Optimize large amounts of repeated data access through layer bulking mode.

6. The method for constructing a graph database according to claim 1, wherein: The specific steps for statistics on the transition graph database are: Use the explain() function to parse how the description statement is transformed into the final set of steps to be executed during compilation; Use profile() to count the time consumed by each step in the execution of the statement and the objects passed.

7. The method for constructing a graph database according to claim 1, wherein: The specific steps for structural adjustment and optimization of the graph database are as follows: Modify the data storage structure of the graph database; Set up and select the appropriate index; Partition the data based on its characteristics; Add caching solution.

8. A system for constructing a graph database, characterized in that: include: A graph database creation module, configured to prepare data according to data query requirements; Insert the chart according to the prepared data and set the chart parameters to obtain the initial chart database; The graph database arrangement module is configured to analyze the attributes of the graph database using a graph search algorithm and a query statement, traverse to obtain all elements and paths; use the query statement to filter, aggregate and optimize the paths to obtain a transition graph database; The graph database optimization module is configured to adjust and optimize the transition graph database to obtain the final graph database, wherein the adjustment and optimization steps include traversal railing optimization of the transition graph database, statistics of the transition graph database, and structural adjustment and optimization of the graph database.

9. A computer-readable storage medium, characterized in that: A plurality of instructions are stored therein, and the instructions are suitable for being loaded by a processor of a terminal device and executing the method for constructing a graph database as described in any one of claims 1-7.

10. A terminal device, characterized in that: It includes a processor and a computer-readable storage medium, the processor is used to implement each instruction; the computer-readable storage medium is used to store multiple instructions, and the instructions are suitable for being loaded by the processor and executed by the method for constructing a graph database described in any one of claims 1-7.

Images