Lightweight knowledge graph construction and analysis method and device, and storage medium

Abstract

The application discloses a kind of lightweight knowledge graph construction and analysis method, device and storage medium, method includes: constructing first graph topology data, first graph topology data includes relationship data;Global uniqueness mapping is carried out to obtain mapping data;Mapping data is converted into analysis domain data, obtain the second graph topology data after conversion, and are divided into different sections;Second graph topology data is converted into hypergraph model, and attribute multilayer management is carried out;According to time and space dimension, different attribute layers in multiple attribute layers are recorded or segmented, and spatiotemporal knowledge graph is constructed;According to the spatiotemporal knowledge graph, the relationship data is analyzed to obtain target analysis result. Through the embodiment of the application, a lightweight knowledge graph construction and analysis scheme can be provided for users when the storage capacity is limited, and the storage resources are fully utilized to quickly and efficiently construct and analyze the knowledge graph, while the performance requirements are considered.

Description

Technical Field

[0001] This invention relates to the field of big data processing technology, and in particular to a lightweight knowledge graph construction and analysis method, apparatus and storage medium. Background Technology

[0002] Knowledge graphs represent the most closely interconnected and widely used application of graph databases. Using graph databases as storage engines, knowledge graphs intelligently process massive amounts of information, forming large-scale knowledge bases to support business applications. In the library and information science field, a knowledge graph is referred to as knowledge domain visualization or a knowledge domain mapping map. It is a series of various graphs that display the development process and structural relationships of knowledge, using visualization techniques to describe knowledge resources and their carriers, and to mine, analyze, construct, draw, and display knowledge and their interrelationships. A graph database, on the other hand, is a data management system designed for efficient storage and retrieval of graph data, using nodes and edges as basic storage units. Unlike graph engines, graph databases are primarily used for Online Transaction Processing (OLTP), performing ACID transaction processing on data. Graph engines are mainly used for Online Analytical Processing (OLAP), performing batch data analysis.

[0003] Currently, mainstream open-source graph databases include Neo4j, ArangoDB, Virtuoso, TigerGraph, RedisGraph, JanusGraph, Giraph, TuGraph, NebulaGraph, Dgraph, and HugeGraph. However, most well-known open-source graph database versions only support single nodes and cannot horizontally scale storage, failing to meet the storage and analysis needs of large-scale graph data. For analytical applications with less stringent time requirements, when computing or memory resources are relatively limited, there is no particularly suitable graph database / graph engine that can efficiently solve the aforementioned knowledge graph construction and analysis problems. In particular, existing technologies are often helpless when memory capacity is much smaller than data capacity. For example, the current prominent contradiction between limited memory resources and large data scale is that processing performance is limited when memory resources cannot accommodate all graph data.

[0004] Therefore, how to make full use of storage resources to quickly and efficiently build and analyze knowledge graphs in graph databases or graph engines with limited storage capacity, while taking performance requirements into account, has become an urgent technical problem to be solved. Summary of the Invention

[0005] The main objective of this invention is to provide a lightweight method, apparatus, and storage medium for knowledge graph construction and analysis, aiming to solve the problem that memory resources cannot accommodate all graph data and processing performance is limited in the prior art.

[0006] To achieve the above objectives, this invention provides a lightweight method for constructing and analyzing knowledge graphs, the method comprising the following steps:

[0007] S1: Construct the first graph topology data, which includes relational data;

[0008] S2: Perform a globally unique mapping on the topology data of the first graph to obtain the mapped data;

[0009] S3: Convert the mapping data into analysis domain data according to the parallel data reading order to obtain the converted second graph topology data, and divide the converted second graph topology data into different segments;

[0010] S4: Convert the transformed second graph topology data into a hypergraph model and establish multiple attribute layers for multi-layer attribute management;

[0011] S5: Record or segment different attribute layers in the multiple attribute layers according to the time and space dimensions to construct a spatiotemporal knowledge graph;

[0012] S6: Perform target analysis on the relational data based on the spatiotemporal knowledge graph to obtain the target analysis results.

[0013] Optionally, step S1 includes the following steps:

[0014] The system quickly reads preset key fields from the relational database in parallel and performs parallel transformations based on the read key fields to obtain the first graph topology data.

[0015] Optionally, the mapping method used in step S2 is either linear mapping or hash mapping.

[0016] Optionally, when converting the mapping data into analysis domain data in step S3, the following steps are also included:

[0017] The mapped data is used as nodes, and a merging and dimensionality reduction operation is performed on the mapped data.

[0018] Among them, the size of the second graph topology data in each group of different segments is the same or approximately the same;

[0019] The process of merging and reducing the dimensions of the mapped data also includes the following steps:

[0020] The edges of the nodes are merged into logical edges to reduce the dimensionality of the edges.

[0021] Optionally, the multiple attribute layers include single-attribute layers and multi-attribute layers, and the management of multiple attribute layers includes the following steps:

[0022] The analysis of the single-attribute layer among the multiple attribute layers includes:

[0023] The single-attribute layer in the multiple attribute layers is subjected to equalization, parallelization, and multi-threading processing based on the improved DFS dual-stack equalization parallelization algorithm and the preset thread data allocation rules.

[0024] The analysis of the multi-attribute layer among the various attribute layers includes:

[0025] The multi-attribute layers among the various attribute layers are processed in parallel according to the mapping simplification parallel algorithm.

[0026] Optionally, step S6 includes the following steps:

[0027] Target analysis is performed on the spatiotemporal knowledge graph and the relational data from the temporal and spatial dimensions to obtain the target analysis results;

[0028] The target analysis method is one or more of the following: differentiation analysis, path analysis, super node analysis, loop analysis, and subgraph analysis.

[0029] Optionally, the method further includes the following steps:

[0030] S7: Backtrack and interpret the target analysis results to obtain the original data of the target analysis results.

[0031] Optionally, step S3 further includes the following steps:

[0032] Determine the storage resource status; if the storage resource is less than a preset threshold, it is considered a lightweight storage resource; otherwise, it is considered a rich storage resource.

[0033] When the storage resources are lightweight, the transformed second graph topology data is segmented and scheduled to divide the transformed second graph topology data into different segments.

[0034] Furthermore, to achieve the above objectives, this invention also proposes a lightweight knowledge graph construction and analysis device, the device comprising:

[0035] The first construction unit is used to construct first graph topology data, which includes relational data.

[0036] The mapping unit is used to perform a globally unique mapping on the topology data of the first graph to obtain the mapping data;

[0037] The second construction unit is used to convert the mapping data into analysis domain data according to the parallel data reading order, obtain the converted second graph topology data, and divide the converted second graph topology data into different segments.

[0038] The layer processing unit is used to convert the transformed second graph topology data into a hypergraph model and to create multiple attribute layers for multi-layer attribute management.

[0039] The third construction unit is used to record or segment different attribute layers in the multiple attribute layers according to the time and space dimensions to construct a spatiotemporal knowledge graph.

[0040] The analysis unit is used to perform target analysis on the relational data based on the spatiotemporal knowledge graph, and to backtrack and interpret the target analysis results.

[0041] Furthermore, to achieve the above objectives, the present invention also proposes a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the lightweight knowledge graph construction and analysis method described above.

[0042] Through this embodiment of the invention, a first graph topology data, including relational data, is constructed; a globally unique mapping is performed on the first graph topology data to obtain mapped data; the mapped data is converted into analysis domain data according to the parallel data reading order to obtain transformed second graph topology data, which is then divided into different segments; the transformed second graph topology data is converted into a hypergraph model, and multiple attribute layers are established for multi-layer attribute management; different attribute layers in the multiple attribute layers are recorded or segmented according to time and space dimensions to construct a spatiotemporal knowledge graph; target analysis is performed on the relational data based on the spatiotemporal knowledge graph to obtain target analysis results. Thus, under conditions of limited storage capacity or much smaller than the data capacity, a lightweight knowledge graph construction and analysis solution can be provided to users within a certain time tolerance range. This fully utilizes storage resources to quickly and efficiently construct and analyze knowledge graphs in graph databases or graph engines, while also considering the overall system performance requirements. Attached Figure Description

[0043] Figure 1 This is a flowchart illustrating the lightweight knowledge graph construction and analysis method provided by the present invention.

[0044] Figure 2This is another flowchart illustrating the lightweight knowledge graph construction and analysis method provided by the present invention.

[0045] Figure 3 This is a flowchart illustrating the lightweight knowledge graph construction method provided by the present invention.

[0046] Figure 4 This is a flowchart illustrating the specific method for judging and processing storage resource conditions provided by the present invention.

[0047] Figure 5 The structural block diagram of the lightweight knowledge graph construction and analysis device provided by the present invention.

[0048] Figure 6 This is a schematic diagram of the lightweight knowledge graph construction and analysis device provided in an embodiment of the present invention.

[0049] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0050] To make the technical problems to be solved, the technical solutions, and the beneficial effects of the present invention clearer and more understandable, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only for explaining the present invention and are not intended to limit the present invention.

[0051] In the following description, the use of suffixes such as "module," "part," or "unit" to denote elements is solely for the purpose of illustrative purposes and has no specific meaning in itself. Therefore, "module," "part," or "unit" may be used interchangeably.

[0052] It should be noted that the terms "first," "second," etc., in the specification, claims, and drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0053] In one embodiment, such as Figure 1 As shown, this invention provides a lightweight knowledge graph construction and analysis method, the method comprising:

[0054] Step S1: Construct the first graph topology data, which includes relational data.

[0055] Step S2: Perform a globally unique mapping on the topology data of the first graph to obtain the mapping data.

[0056] Step S3: Convert the mapping data into analysis domain data according to the parallel reading order of the data to obtain the converted second graph topology data, and divide the converted second graph topology data into different segments.

[0057] Step S4: Convert the transformed second graph topology data into a hypergraph model and establish multiple attribute layers for multi-layer attribute management.

[0058] Step S5: Record or segment different attribute layers in the multiple attribute layers according to time and space dimensions to construct a spatiotemporal knowledge graph.

[0059] Step S6: Perform target analysis on the relational data based on the spatiotemporal knowledge graph to obtain the target analysis results.

[0060] Through the embodiments of the present invention, under the condition that the storage capacity is limited or much smaller than the data capacity, a lightweight knowledge graph construction and analysis solution can be provided to users under certain time tolerance conditions. It makes full use of storage resources to quickly and efficiently construct and analyze knowledge graphs in graph databases or graph engines, while taking into account the overall performance requirements of the system.

[0061] In order to make full use of storage resources to quickly and efficiently construct and analyze knowledge graphs in graph databases or graph engines, given the limited storage capacity or the fact that storage capacity is often much smaller than the data capacity, this invention proposes a technical solution.

[0062] In step S1, preset key fields can be quickly and in parallel read from a traditional relational database. Then, the read key fields can be transformed in parallel according to actual needs to construct graph topology data, that is, the first graph topology data can be obtained. ， Among them, the first graph topology data includes at least relational data, which is data in a traditional relational database. For example, preset key fields can be read from an MPP (Massively Parallel Processor) database using parallel SQL queries. To protect data privacy, the key fields are further transformed in parallel.

[0063] In step S2, after obtaining the first graph topology data, a globally unique mapping can be performed on the first graph topology data to obtain mapped data. A globally unique mapping results in the same outcome regardless of time or location, such as in memory or on a storage disk. The mapping method used for the first graph topology data can be a linear mapping or a hash mapping. For example, the uniqueness of a name and ID number can be used to perform an MD5 hash, resulting in a 16-byte or 32-byte MD5 hash string.

[0064] Of course, it should be noted that when performing a globally unique mapping on the first graph's topological data, there may be cases where the global uniqueness requirement is not met. This is mainly because the mapping function may have different inputs but the same output. For example, MD5 may experience hash collisions, meaning two different inputs can produce the same output. In such cases, further extended processing is needed. For instance, collision information can be compared to ensure a globally unique mapping and obtain the mapped data. Specifically, when global uniqueness is not met, the mapping relationship between values ​​and (multiple) sources can be stored to build a fault-tolerance mechanism. If different sources are found to produce the same mapped value during the addition process, a local mapping table is added for upgrade comparison information. For example, when f(x1) = y and f(x2) = y, a local mapping table of y → x1, x2 can be stored, thus enabling special processing of value encoding bit extension when the mapping is not unique. In this way, extending the encoding by one bit to mark whether there is a non-unique source mapping table also facilitates subsequent target analysis or backtracking interpretation. After extending the number of bits in the encoding of the non-unique mapping value, global uniqueness can be satisfied, and global unique mapping can be performed to obtain the mapped data.

[0065] In step S3, to facilitate subsequent grouping, segmentation, and processing of the data source, different data (such as transaction domain data) can be quickly and concurrently converted into analysis domain data according to the parallel reading order based on the mapping data results, thereby obtaining the converted second graph topology data. The converted second graph topology data can be divided into different segments. The converted second graph topology data in each segment is basically the same or approximately the same size, which is convenient for lightweight, low-memory processing or distributed, fast parallel processing.

[0066] Optionally, when converting the mapping data into analysis domain data, the mapping data can be merged and dimensionality reduced to decrease the complexity of the graph. Specifically, the mapping data can be treated as nodes, belonging to the original nodes. By merging and reducing the mapping data into logical nodes, the dimensionality reduction of the points can be achieved, and the dimensionality-reduced mapping relationship between the original nodes and the logical nodes can be constructed. That is, different known relationships can be formed into a subgraph, and each subgraph has only a representative and unique mapping value externally. For example, a family's external relationship is attached to the head of household, thereby reducing the number of composite nodes, reducing space occupancy, and significantly reducing the analysis processing time and improving analysis efficiency when analyzing the topological data of the transformed second graph.

[0067] For example, in family relationships, there are ten types of people related to the head of household, specifically including the head of household, the head of household's spouse, the head of household's children, the head of household's parents, the head of household's grandparents, the head of household's maternal grandparents, the head of household's siblings, the head of household's collateral relatives, and other relatives. These various relationships in the household registration book can be merged and simplified, with the head of household representing a family, and can be represented by the head of household's name and ID number.

[0068] For example, in corporate relationships, there are various types of relationships between legal persons, including the legal person himself / herself, employment relationships, subsidiary relationships, branch relationships, etc. These relationships can be merged and simplified, and represented by the parent company's organization code.

[0069] While performing dimensionality reduction on nodes, different data (such as transaction domain data) can be transformed into analysis domain data by merging the edges of nodes into logical edges, thus reducing graph complexity and improving analysis efficiency. In other words, dimensionality reduction operations can be performed on the edges of nodes; the original edges can be merged into logical edges, and a dimensionality reduction mapping relationship between the original edges and logical edges can be constructed. For example, each income transaction of parents and children can be grouped into a family income attribute, represented by the head of household. Therefore, all external income is mapped to the head of household's income. If both parents work at the same company, multiple income transactions from the same company are grouped into one logical edge for the head of household's income. Similarly, multiple purchases by the family from the same store are grouped into another logical edge for the head of household's purchases.

[0070] In step S4, after the processing in step S3, the transformed second graph topology data can be converted into a hypergraph model for processing. It should be noted that the specific conversion process is existing technology and will not be described in detail in this embodiment.

[0071] After conversion to a hypergraph model, multiple attribute layers can be created for multi-layer attribute management. For example, a family's income information can be used as one attribute layer, consumption information as another, and additional social information as a new attribute layer, thus constructing multiple attribute layers.

[0072] Since the multiple attribute layers can include single attribute layers and multi-attribute layers, the management of multiple attribute layers can include analyzing single attribute layers among the multiple attribute layers, and can also include analyzing multi-attribute layers among the multiple attribute layers.

[0073] When analyzing a single-attribute layer in the multiple attribute layers, parallel multi-threaded processing of equalization can be performed on the single-attribute layer in the multiple attribute layers according to the improved DFS dual-stack equalization parallel algorithm and the preset thread data allocation rule. By using the improved DFS dual-stack equalization parallel algorithm, parallel multi-threaded processing of equalization can be achieved, thereby further improving the performance of the data processing system. Moreover, by using the improved DFS dual-stack equalization parallel algorithm, overflow caused by recursive callbacks can be avoided under the condition of storage resource limitation, and at the same time, the processing performance of super nodes can be improved. However, the same type of algorithms do not achieve parallel processing. They cannot make full use of computing resources under the condition of limited memory resources to improve processing performance as much as possible. For example, the existing DFS (Depth-First-Search) algorithm does not achieve equalization parallel ability either. The existing DFS algorithm is an algorithm used to traverse or search a tree or graph. It traverses the nodes of the tree along the depth of the tree and searches the branches of the tree as deep as possible. When all the edges of a node have been explored, the search will backtrack to the starting node of the edge where the node was found. This process continues until all nodes reachable from the source node have been discovered. If there are still undiscovered nodes, one of them is selected as the source node and the above process is repeated. The entire process is repeated until all nodes have been visited. It belongs to blind search.

[0074] Specifically, the improved DFS dual-stack equalization parallel algorithm achieves parallel computing ability compared with the existing DFS algorithm, realizes balanced thread allocation, and the most important difference is that the deployment strategy therein can make full use of the computing resource advantages of multi-core or distributed computing, while reducing the number of allocations and further enhancing the performance. Specifically, the deployment strategy is the preset thread data allocation rule, that is, the preset thread data allocation rule is to make full use of computer resources, reduce the number of thread allocations, and further improve the analysis and processing performance under the condition of limited memory resources.

[0075] Specifically, the preset thread data allocation rule can be as follows:

[0076] I. Initial allocation rule:

[0077] Allocate according to the number N of starting topology files and the number M of thread pools.

[0078] 1. If N = 1 and the out-degree of the topology file is K, the allocation rule is as follows:

[0079] 1) If K < M, then allocate K to the first K threads, and the allocation ends. The remaining idle threads are re-allocated according to the situation of the next node.

[0080] 2) If K ≥ M, then allocate K evenly to M threads, and the last thread is allocated the least number of out-degrees.

[0081] 2. If N < M, the allocation rules are as follows:

[0082] 1) Allocate N - 1 files to the first N - 1 threads.

[0083] 2) Allocate the Nth file according to the rule in item 1.

[0084] 3. If N ≥ M, the allocation rules are as follows:

[0085] 1) Initial allocation rule: Each thread is allocated one topology file.

[0086] 2) Secondary allocation rule: If a thread finishes execution, take the next topology file until all topology files are allocated.

[0087] II. Re - allocation rules when there are idle threads:

[0088] 1. When an idle thread reports its idle state to the main thread, it is in the waiting stage.

[0089] 2. When the main thread detects that there are idle threads, it sends a broadcast to each thread, reporting the depth of the current processing stack and the out - degree of the pending tasks in the processing stack.

[0090] 3. The main thread selects the thread with the shallowest stack and the most out - degrees for average allocation according to the processing status of each thread.

[0091] When analyzing the multi - attribute layers in the multiple attribute layers, the multi - attribute layers in the multiple attribute layers can be processed in parallel according to the MapReduce parallel algorithm to make full use of the storage space. It should be noted that the MapReduce parallel algorithm is an existing design, and this embodiment will not elaborate on it here. The MapReduce parallel algorithm used in this embodiment has no difference from the MapReduce parallel algorithm in the prior art, except for the application innovation, which is used in the hypergraph model processing to improve the performance of hypergraph processing, make full use of the computing power, especially when association analysis of multiple attribute layers is required.

[0092] In step S5, different attribute layers in the multiple attribute layers described in step S4 can be recorded or segmented according to time and space dimensions to construct a spatiotemporal knowledge graph. It is understood that the aforementioned spatial dimension is not limited to natural three-dimensional space, but can also be an analytical domain mapping space, such as the value domain space in source-to-value mapping. For example, the time dimension of the multiple attribute layers can be stored and segmented according to a date folder format (yyyymmdd) or a filename prefix encoding format; different business attribute layers can be stored in folders or merged bit by bit, for example, one byte can represent a maximum of eight business attributes, thus further reducing storage space; then, the data of multiple attribute layers can be segmented and stored according to the volume / surface / line of natural three-dimensional space, or segmented and stored according to the mapping encoded value domain space.

[0093] In step S6, as people's understanding of the objective world deepens and the amount of information and knowledge continuously increases, the content of the knowledge graph also keeps pace with the times, constantly changing and iterating, adding new knowledge and deleting outdated knowledge. Therefore, after constructing the spatiotemporal knowledge graph, subsequent target analysis can be performed on the spatiotemporal knowledge graph and the relational data from the time and space dimensions. For example, analysis of additions, disappearances, or changes can be performed to discover possible abnormal information, thereby obtaining target analysis results. Specifically, the target analysis method can be one or more of the following: differentiation analysis, path analysis, super node analysis, loop analysis, and subgraph analysis. Among them, path analysis can also include full path analysis, path degree analysis, and reverse path analysis. Since the above analysis methods are existing technologies, they will not be described in detail in this embodiment.

[0094] Through the embodiments of the present invention, under conditions where storage capacity is limited or much smaller than data capacity, a lightweight knowledge graph construction and analysis solution can be provided to users within a certain time tolerance period. This solution fully utilizes storage resources to quickly and efficiently construct and analyze knowledge graphs in graph databases or graph engines, while also considering the overall system performance requirements. In practical applications, it can be used for geographic entity relationship analysis, epidemic transmission chain analysis, and underground pipeline network analysis, such as pipe burst analysis; these will not be elaborated further in this embodiment.

[0095] It should be noted that this application is not limited to directed or undirected graphs, and can also perform reverse analysis.

[0096] Please refer to further information. Figure 2 ,like Figure 2 As shown, this invention provides a lightweight knowledge graph construction method, in Figure 1 As shown in the steps, the method includes step S7, that is, the method includes:

[0097] Step S1: Construct the first graph topology data, which includes relational data.

[0098] Step S2: Perform a globally unique mapping on the topology data of the first graph to obtain the mapping data.

[0099] Step S3: Convert the mapping data into analysis domain data according to the parallel reading order of the data to obtain the converted second graph topology data, and divide the converted second graph topology data into different segments.

[0100] Step S4: Convert the transformed second graph topology data into a hypergraph model and establish multiple attribute layers for multi-layer attribute management.

[0101] Step S5: Record or segment different attribute layers in the multiple attribute layers according to time and space dimensions to construct a spatiotemporal knowledge graph.

[0102] Step S6: Perform target analysis on the relational data based on the spatiotemporal knowledge graph to obtain the target analysis results.

[0103] Step S7: Backtrack and interpret the target analysis results to obtain the original data of the target analysis results.

[0104] Because steps S1 to S6 in this embodiment are... Figure 1 Steps S1 to S6 are the same, and will not be repeated in this embodiment.

[0105] It should be noted that the backtracking interpretation is based on the fact that a merging and dimensionality reduction process was performed in step S3 of this embodiment. Without backtracking interpretation, the original data cannot be restored. Figure 2 As shown. If the merging and dimensionality reduction process is not performed in step S3, the original data can be obtained directly after obtaining the target analysis results, without the need for backtracking interpretation.

[0106] In step S6, since the merging and dimensionality reduction process was performed in step S3, the input original node or original edge parameters need to be processed during target analysis. Through the dimensionality reduction mapping relationship between the original node and the logical node, or the dimensionality reduction mapping relationship between the original edge and the logical edge, they are uniformly transformed into logical nodes or logical edges before target analysis is performed. The result of target analysis is a set of results containing logical nodes and logical edges.

[0107] In step S7, after obtaining the target analysis results, since the nodes and edges have already undergone dimensionality reduction, and a hypergraph has been formed from multi-attribute layers, backtracking is required to restore the detailed relationships. Specifically, by backtracking and interpreting the target analysis results, the original data of the target analysis results can be obtained. The backtracking and interpretation process involves filtering out those original edges and original nodes that are not directly associated with the original nodes based on the input original node / original edge parameters, the dimensionality reduction mapping relationship between original nodes and logical nodes, and the dimensionality reduction mapping relationship between original edges and logical edges, to obtain the actual analysis results.

[0108] For backtracking interpretation, for example, if household head A (logical node) purchases goods from company B (logical node), backtracking interpretation will show that household member A1 actually purchased goods from subsidiary B1 of company B, then the backtracking result is A1→B1.

[0109] It should be noted that logical nodes possess the basic information of the original nodes and also carry different business extension attributes; logical edges also have two meanings: first, the nodes of a logical edge refer to logical nodes; second, a logical edge contains multiple business attribute layer relationships. This embodiment only selects the attribute layers of relevant businesses for analysis. Even within the same business attribute layer, there may be multiple relationships, just as the same vehicle may repeatedly travel on a road; different business attribute layers can be understood as different brands and models of vehicles representing different businesses, and multiple business interactions occurring on the same road.

[0110] Please refer to further information. Figure 3 ,like Figure 3 As shown, this invention provides a lightweight knowledge graph construction method, the method comprising:

[0111] Step S301: Construct first graph topology data, which includes relational data.

[0112] Step S302: Perform a globally unique mapping on the topology data of the first graph to obtain the mapping data.

[0113] Step S303: Convert the mapping data into analysis domain data according to the parallel reading order of the data. When the mapping data is converted into analysis domain data, the mapping data is used as a node. Merge and reduce the dimension of the mapping data and merge the edges of the nodes into logical edges to reduce the dimension of the nodes, so as to obtain the converted second graph topology data.

[0114] Step S304: When the storage resource is a lightweight storage resource, the transformed second graph topology data is segmented and scheduled to divide the transformed second graph topology data into different groups; wherein, the size of the second graph topology data in each group is the same or approximately the same.

[0115] Step S305: Convert the transformed second graph topology data into a hypergraph model and establish multiple attribute layers for multi-layer attribute management.

[0116] Step S306: Record or segment different attribute layers in the multiple attribute layers according to time and space dimensions to construct a spatiotemporal knowledge graph.

[0117] It should be noted that steps S301 to S303 are the same as steps S1 to S3, and steps S305 to S306 are the same as steps S4 to S5; these will not be repeated here in this embodiment. The difference lies only in step S304. To adapt to lightweight processing with limited memory and computing resources under normal conditions, when memory resources are limited and it is impossible to load all the topology data into memory, when it is determined that the storage resources are lightweight, the transformed second graph topology data is segmented and scheduled, dividing the transformed second graph topology data into different groups; wherein the size of the second graph topology data in each group is the same or approximately the same. However, this still requires sufficient low-speed (cache) storage resources.

[0118] For detailed information on the assessment and handling of storage resource status, please refer to the relevant documentation. Figure 4 ,like Figure 4 As shown, it includes:

[0119] Step S401: Determine the storage resource status and determine whether the storage resources are less than a preset threshold.

[0120] If so, it belongs to lightweight storage resources, proceed to step S402; otherwise, it belongs to abundant storage resources, indicating that the memory resources are large enough to load all topology data into memory, providing processing performance. Specifically, the converted second graph topology data can be elastically expanded. It should be noted that this embodiment discusses a lightweight storage resource solution; abundant storage resources are a common situation in the prior art, and will not be described in detail here.

[0121] Step S402: Perform segmented scheduling processing on the transformed second graph topology data to divide the transformed second graph topology data into different groups.

[0122] It should be noted that the preset threshold can be set, but this embodiment does not impose any restrictions. Segmented scheduling processing is existing technology and will not be described in detail here.

[0123] Furthermore, embodiments of the present invention also propose a lightweight knowledge graph construction and analysis device, referring to... Figure 5 The lightweight knowledge graph construction and analysis device includes:

[0124] The first construction unit 501 is used to construct the first graph topology data, which includes relational data.

[0125] The mapping unit 502 is used to perform a globally unique mapping on the topology data of the first graph to obtain the mapping data.

[0126] The second construction unit 503 is used to convert the mapping data into analysis domain data according to the parallel reading order of the data, obtain the converted second graph topology data, and divide the converted second graph topology data into different segments.

[0127] The layer processing unit 504 is used to convert the transformed second graph topology data into a hypergraph model and establish multiple attribute layers for multi-layer attribute management.

[0128] The third construction unit 505 is used to record or segment different attribute layers in the multiple attribute layers according to the time and space dimensions to construct a spatiotemporal knowledge graph.

[0129] Analysis unit 506 is used to perform target analysis on the relational data based on the spatiotemporal knowledge graph to obtain target analysis results.

[0130] Through the embodiments of the present invention, under the condition that the storage capacity is limited or much smaller than the data capacity, a lightweight knowledge graph construction and analysis solution can be provided to users under certain time tolerance conditions. It makes full use of storage resources to quickly and efficiently construct and analyze knowledge graphs in graph databases or graph engines, while taking into account the overall performance requirements of the system.

[0131] It should be noted that each module or unit in the above system can be used to implement each step in the above method and achieve the corresponding technical effect. This embodiment will not elaborate further here.

[0132] Reference Figure 6 , Figure 6 This is a schematic diagram of a lightweight knowledge graph construction and analysis device provided in an embodiment of the present invention.

[0133] like Figure 6As shown, this lightweight knowledge graph construction and analysis device may include: a processor 1001, such as a CPU; a communication bus 1002; a user interface 1003; a network interface 1004; and a memory 1005. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen or an input unit such as a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as Wi-Fi, 4G, or 5G). The memory 1005 may be high-speed RAM or stable non-volatile memory, such as disk storage. Optionally, the memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0134] Those skilled in the art will understand that Figure 6 The structure shown does not constitute a limitation on lightweight knowledge graph construction and analysis apparatus, and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0135] like Figure 6 As shown, the memory 1005, which serves as a computer storage medium, may include an operating system, a network communication module, a user interface module, and a lightweight knowledge graph construction and analysis program.

[0136] exist Figure 6 In the lightweight knowledge graph construction and analysis device shown, the network interface 1004 is mainly used for data communication with an external network; the user interface 1003 is mainly used for receiving user input commands; the lightweight knowledge graph construction and analysis device calls the lightweight knowledge graph construction and analysis program stored in the memory 1005 through the processor 1001. This lightweight knowledge graph construction and analysis device is used to execute lightweight knowledge graph construction and analysis methods, and may specifically include a PC, server, server cluster, or computer product with lightweight knowledge graph construction and analysis functions and a big data platform.

[0137] Specifically, the lightweight knowledge graph construction and analysis device calls the lightweight knowledge graph construction and analysis program stored in the memory 1005 through the processor 1001 and performs the following operations:

[0138] S1: Construct the first graph topology data, which includes relational data;

[0139] S2: Perform a globally unique mapping on the topology data of the first graph to obtain the mapped data;

[0140] S3: Convert the mapping data into analysis domain data according to the parallel data reading order to obtain the converted second graph topology data, and divide the converted second graph topology data into different segments;

[0141] S4: Convert the transformed second graph topology data into a hypergraph model and establish multiple attribute layers for multi-layer attribute management;

[0142] S5: Record or segment different attribute layers in the multiple attribute layers according to the time and space dimensions to construct a spatiotemporal knowledge graph;

[0143] S6: Perform target analysis on the relational data based on the spatiotemporal knowledge graph to obtain the target analysis results.

[0144] Optionally, step S1 includes the following steps:

[0145] The system quickly reads preset key fields from the relational database in parallel and performs parallel transformations based on the read key fields to obtain the first graph topology data.

[0146] Optionally, the mapping method used in step S2 is either linear mapping or hash mapping.

[0147] Optionally, when converting the mapping data into analysis domain data in step S3, the following steps are also included:

[0148] The mapped data is used as nodes, and a merging and dimensionality reduction operation is performed on the mapped data.

[0149] Among them, the size of the second graph topology data in each group of different segments is the same or approximately the same;

[0150] The process of merging and reducing the dimensions of the mapped data also includes the following steps:

[0151] The edges of the nodes are merged into logical edges to reduce the dimensionality of the edges.

[0152] Optionally, the multiple attribute layers include single-attribute layers and multi-attribute layers, and the management of multiple attribute layers includes the following steps:

[0153] The analysis of the single-attribute layer among the multiple attribute layers includes:

[0154] The single-attribute layer in the multiple attribute layers is subjected to equalization, parallelization, and multi-threading processing based on the improved DFS dual-stack equalization parallelization algorithm and the preset thread data allocation rules.

[0155] The analysis of the multi-attribute layer among the various attribute layers includes:

[0156] The multi-attribute layers among the various attribute layers are processed in parallel according to the mapping simplification parallel algorithm.

[0157] Optionally, step S6 includes the following steps:

[0158] Target analysis is performed on the spatiotemporal knowledge graph and the relational data from the temporal and spatial dimensions to obtain the target analysis results;

[0159] The target analysis method is one or more of the following: differentiation analysis, path analysis, super node analysis, loop analysis, and subgraph analysis.

[0160] Optionally, the method further includes the following steps:

[0161] S7: Backtrack and interpret the target analysis results to obtain the original data of the target analysis results.

[0162] Optionally, step S3 further includes the following steps:

[0163] Determine the storage resource status; if the storage resource is less than a preset threshold, it is considered a lightweight storage resource; otherwise, it is considered a rich storage resource.

[0164] When the storage resources are lightweight, the transformed second graph topology data is segmented and scheduled to divide the transformed second graph topology data into different segments.

[0165] Through the embodiments of the present invention, under the condition that the storage capacity is limited or much smaller than the data capacity, a lightweight knowledge graph construction and analysis solution can be provided to users under certain time tolerance conditions. It makes full use of storage resources to quickly and efficiently construct and analyze knowledge graphs in graph databases or graph engines, while taking into account the overall performance requirements of the system.

[0166] Furthermore, embodiments of the present invention also propose a computer-readable storage medium storing a lightweight knowledge graph construction and analysis program. When executed by a processor, the lightweight knowledge graph construction and analysis program performs the following operations:

[0167] S1: Construct the first graph topology data, which includes relational data;

[0168] S2: Perform a globally unique mapping on the topology data of the first graph to obtain the mapped data;

[0169] S3: Convert the mapping data into analysis domain data according to the parallel data reading order to obtain the converted second graph topology data, and divide the converted second graph topology data into different segments;

[0170] S4: Convert the transformed second graph topology data into a hypergraph model and establish multiple attribute layers for multi-layer attribute management;

[0171] S5: Record or segment different attribute layers in the multiple attribute layers according to the time and space dimensions to construct a spatiotemporal knowledge graph;

[0172] S6: Perform target analysis on the relational data based on the spatiotemporal knowledge graph to obtain the target analysis results.

[0173] Optionally, step S1 includes the following steps:

[0174] The system quickly reads preset key fields from the relational database in parallel and performs parallel transformations based on the read key fields to obtain the first graph topology data.

[0175] Optionally, the mapping method used in step S2 is either linear mapping or hash mapping.

[0176] Optionally, when converting the mapping data into analysis domain data in step S3, the following steps are also included:

[0177] The mapped data is used as nodes, and a merging and dimensionality reduction operation is performed on the mapped data.

[0178] Among them, the size of the second graph topology data in each group of different segments is the same or approximately the same;

[0179] The process of merging and reducing the dimensions of the mapped data also includes the following steps:

[0180] The edges of the nodes are merged into logical edges to reduce the dimensionality of the edges.

[0181] Optionally, the multiple attribute layers include single-attribute layers and multi-attribute layers, and the management of multiple attribute layers includes the following steps:

[0182] The analysis of the single-attribute layer among the multiple attribute layers includes:

[0183] The single-attribute layer in the multiple attribute layers is subjected to equalization, parallelization, and multi-threading processing based on the improved DFS dual-stack equalization parallelization algorithm and the preset thread data allocation rules.

[0184] The analysis of the multi-attribute layer among the various attribute layers includes:

[0185] The multi-attribute layers among the various attribute layers are processed in parallel according to the mapping simplification parallel algorithm.

[0186] Optionally, step S6 includes the following steps:

[0187] Target analysis is performed on the spatiotemporal knowledge graph and the relational data from the temporal and spatial dimensions to obtain the target analysis results;

[0188] The target analysis method is one or more of the following: differentiation analysis, path analysis, super node analysis, loop analysis, and subgraph analysis.

[0189] Optionally, the method further includes the following steps:

[0190] S7: Backtrack and interpret the target analysis results to obtain the original data of the target analysis results.

[0191] Optionally, step S3 further includes the following steps:

[0192] Determine the storage resource status; if the storage resource is less than a preset threshold, it is considered a lightweight storage resource; otherwise, it is considered a rich storage resource.

[0193] When the storage resources are lightweight, the transformed second graph topology data is segmented and scheduled to divide the transformed second graph topology data into different segments.

[0194] Through the embodiments of the present invention, under the condition that the storage capacity is limited or much smaller than the data capacity, a lightweight knowledge graph construction and analysis solution can be provided to users under certain time tolerance conditions. It makes full use of storage resources to quickly and efficiently construct and analyze knowledge graphs in graph databases or graph engines, while taking into account the overall performance requirements of the system.

[0195] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0196] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0197] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, controller, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0198] The above are merely preferred embodiments of the present invention and do not limit the scope of the patent. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.

Claims

1. A lightweight knowledge graph construction and analysis method, characterized in that, The method includes the following steps: S1: Construct the first graph topology data, which includes relational data; S2: Perform a globally unique mapping on the topology data of the first graph to obtain the mapped data; S3: Based on the parallel data reading order, the mapping data is converted into analysis domain data after merging and dimensionality reduction to obtain the transformed second graph topology data, and the transformed second graph topology data is divided into different segments; wherein, the size of the second graph topology data in each segment in different segments is the same or approximately the same; the merging and dimensionality reduction includes dimensionality reduction by merging the mapping data as nodes into logical nodes to form points, and dimensionality reduction by merging the edges of the nodes into logical edges to form edges; Step S3 further includes the following steps: Determine the storage resource status; if the storage resource is less than a preset threshold, it is considered a lightweight storage resource; otherwise, it is considered a rich storage resource. When the storage resource is a lightweight storage resource, the transformed second graph topology data is segmented and scheduled to divide the transformed second graph topology data into different segments. S4: Convert the second graph topology data after it has been divided into different groups into a hypergraph model, and establish multiple attribute layers for multi-layer attribute management; S5: Record or segment different attribute layers in the multiple attribute layers according to the time and space dimensions to construct a spatiotemporal knowledge graph; S6: Perform target analysis on the relational data based on the spatiotemporal knowledge graph to obtain the target analysis results.

2. The method according to claim 1, characterized in that, Step S1 includes the following steps: The system quickly reads preset key fields from the relational database in parallel and performs parallel transformations based on the read key fields to obtain the first graph topology data.

3. The method according to claim 1, characterized in that, The mapping method used in step S2 is either linear mapping or hash mapping.

4. The method according to claim 1, characterized in that, The multiple attribute layers include single-attribute layers and multi-attribute layers. The management of multiple attribute layers includes the following steps: The analysis of the single-attribute layer among the multiple attribute layers includes: The single-attribute layer in the multiple attribute layers is subjected to equalization, parallelization, and multi-threading processing based on the improved DFS dual-stack equalization parallelization algorithm and the preset thread data allocation rules. The analysis of the multi-attribute layer among the various attribute layers includes: The multi-attribute layers among the various attribute layers are processed in parallel according to the mapping simplification parallel algorithm.

5. The method according to claim 1, characterized in that, Step S6 includes the following steps: Target analysis is performed on the spatiotemporal knowledge graph and the relational data from the temporal and spatial dimensions to obtain the target analysis results; The target analysis method is one or more of the following: differentiation analysis, path analysis, super node analysis, loop analysis, and subgraph analysis.

6. The method according to claim 1, characterized in that, The method further includes the following steps: S7: Backtrack and interpret the target analysis results to obtain the original data of the target analysis results.

7. A lightweight knowledge graph construction and analysis device, characterized in that, The device includes: The first construction unit is used to construct first graph topology data, which includes relational data. The mapping unit is used to perform a globally unique mapping on the topology data of the first graph to obtain the mapping data; The second construction unit is used to convert the mapping data into merged and dimensionality-reduced analysis domain data according to the parallel data reading order, to obtain the transformed second graph topology data, and to divide the transformed second graph topology data into different segments; wherein, the size of the second graph topology data in each segment in the different segments is the same or approximately the same; the merging and dimensionality reduction includes dimensionality reduction by merging the mapping data as nodes into logical nodes to form points, and dimensionality reduction by merging the edges of the nodes into logical edges to form edges; the second construction unit is also used to: determine the storage resource status; wherein, if the storage resource is less than a preset threshold, it belongs to lightweight storage resources; otherwise, it belongs to abundant storage resources; when the storage resource status belongs to lightweight storage resources, the transformed second graph topology data is segmented and scheduled to divide the transformed second graph topology data into different segments; The layer processing unit is used to convert the second graph topology data, which has been divided into different segments, into a hypergraph model and to establish multiple attribute layers for multi-layer attribute management. The third construction unit is used to record or segment different attribute layers in the multiple attribute layers according to the time and space dimensions to construct a spatiotemporal knowledge graph. The analysis unit is used to perform target analysis on the relational data based on the spatiotemporal knowledge graph, and to backtrack and interpret the target analysis results.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the lightweight knowledge graph construction and analysis method according to any one of claims 1 to 6.

Images