Efficient topology and attribute storage method and device for RAG knowledge graph

Abstract

The invention discloses an efficient topology and attribute storage method and device oriented to an RAG knowledge graph, graph topology and attribute data are separately stored, so that the size of a storage file of an obtained topological structure does not exceed the maximum use space of a memory, and therefore, when topological structure traversal is carried out, the storage efficiency of the storage file is greatly improved. According to the method, the topological structure can be completely read into the memory, so that the problem that the disks need to jump one by one along a chain structure in the traditional technology is avoided, and the topological access efficiency is improved; meanwhile, initial offset and length information during storage of variable-length attribute data are maintained in fixed-length attribute records, and the fixed-length attributes are sequentially stored according to IDs and are loaded through memory mapping, so that direct access can be realized within O (1) time without deserialization, the variable-length attributes are accessed through offset indexes, and the access efficiency is improved. The problem of multi-layer structure analysis is also avoided; therefore, the graph data access performance and the system concurrency capability are remarkably improved.

Description

Technical Field

[0001] This invention relates to the field of graph database and graph computing engine technology, specifically to an efficient method and apparatus for storing topology and attributes for RAG knowledge graphs. Background Technology

[0002] Currently, in graph databases or graph analysis systems, common graph data storage methods often combine graph topology information with node or edge attribute data. For example, in Neo4j, each node record contains a linked list pointer to an edge, and each edge record stores the source and destination nodes, as well as a pointer to the next edge, thus forming a chained adjacency structure. That is, the attribute data of nodes and edges is usually stored in the form of attribute linked lists. Each attribute record stores key-value pairs of one or more fields. Fixed-length attributes (such as integers and floating-point numbers) can be stored directly in the record, while variable-length attributes (such as strings) are offset to another dynamic storage area. This type of chained structure has a certain degree of flexibility, but it also has the following problems: (1) Because the topology and attribute information are stored together in the form of a linked list, the stored linked list occupies a large space and cannot be fully read into memory. As a result, when traversing the topology, it is necessary to jump to the disk one by one along the linked structure. Especially when the node degree is high or the graph structure is complex, a large number of random accesses will be generated, thereby reducing access efficiency.

[0003] (2) Attribute data is scattered across multiple disk pages. Obtaining the complete attribute value of a node often requires multiple jumps, which affects access locality.

[0004] (3) Although fixed-length and variable-length attributes are logically separated, they are still stored in a unified chain structure, which occupies a large space, exceeding the maximum memory usage space. Therefore, it is difficult to achieve mmap mapping or batch loading optimization. Among them, although some graph computing frameworks (such as PowerGraph and GraphX) use compressed adjacency lists (CSR / CSC) and other optimization forms for topology, their attribute data are still usually managed independently in columnar or row-based manner, and do not make full use of memory mapping capabilities for zero-copy access. Therefore, most systems sacrifice read performance in storage model, which is especially unsuitable for static graph scenarios with high query speed requirements.

[0005] Therefore, existing graph database systems and graph computing engines still have bottlenecks in terms of topology access efficiency, attribute access overhead, and data locality. There is a lack of a graph data storage solution that is structurally clear, fast to access, supports mmap loading, and is suitable for static graph scenarios. Summary of the Invention

[0006] The technical problem to be solved by this invention is the problem of graph data storage in graph databases or graph analysis systems. The purpose is to provide an efficient topology and attribute storage method and device for RAG knowledge graphs, which solves the problems of low topology access efficiency, multiple jumps required for attribute access, difficulty in achieving mmap mapping and inability to perform zero-copy access in traditional technologies.

[0007] This invention is achieved through the following technical solution: Firstly, an efficient topology and attribute storage method for RAG knowledge graphs is provided, including: Parse the RAG knowledge graph to obtain the graph topology information, as well as the attribute information of each node and edge in the RAG knowledge graph; The graph topology information is stored to obtain a graph topology file; The attribute information of each node and edge is divided into fixed-length attribute data and variable-length attribute data; The variable length attribute data of each node and the variable length attribute data of each edge are stored in the node variable length attribute file and the edge variable length attribute file respectively, so as to obtain the first starting storage position and the first storage length of the variable length attribute data of each node, and the second starting storage position and the second storage length of the variable length attribute data of each edge respectively. Based on the ID, first storage length, first starting storage position and fixed length attribute data of each node, a fixed length structure of each node is generated, and based on the ID, second storage length, second starting storage position and fixed length attribute data of each edge, a fixed length structure of each edge is generated. According to the ID order of each node, the fixed-length structure of each node is stored in the node fixed-length attribute file, and according to the ID order of each edge, the fixed-length structure of each edge is stored in the edge fixed-length attribute file. After storage, the node fixed-length attribute file and the edge fixed-length attribute file are mapped to the process virtual memory through the mmap mechanism, so as to complete the storage of the topology and attribute information of the RAG knowledge graph.

[0008] Based on the above-disclosed content, after parsing the RAG knowledge graph and obtaining the graph topology information and the attribute information of each node and edge in the graph, this invention stores the graph topology information and attribute information separately. In this way, the traditional common storage is divided into topology storage and attribute information storage, which can refine the storage and make the storage occupy no more than the maximum memory space. Based on this, when traversing the topology, the entire topology can be read into memory, thereby avoiding the problem of traditional technology requiring disk jumps one by one along the chain structure, thus improving the efficiency of topology access.

[0009] Simultaneously, when storing attribute information, a fixed-length structure is generated based on the starting position and length of the variable-length attribute, combined with the corresponding fixed-length attribute. The fixed-length structure of the node is stored in the node fixed-length attribute file, and the fixed-length structure of the edge is stored in the edge fixed-length attribute file, according to the ID order of the node and edge. Thus, the fixed-length attribute record maintains the starting offset and length information corresponding to the storage of the variable-length attribute data. This allows for easy access by first locating the fixed-length attribute structure and then using its internal offset to extract the complete variable-length attribute data from the variable-length attribute file. Based on this, the design avoids chain-like traversal, simplifies the access process, and only requires a single jump to obtain the complete variable-length attribute value, thereby improving data decoding efficiency.

[0010] In addition, the file is mapped to the process virtual memory through the mmap mechanism, which enables zero-copy access to the data file; at the same time, separate storage ensures that the size of the attribute information does not exceed the maximum memory space, and the aforementioned attribute storage mechanism allows the entire attribute information to be read with only a fixed-length file. Based on this, the size of the attribute file that needs to be mapped by mmap can be further reduced, thereby making it very easy to implement mmap mapping for the attribute file of the present invention.

[0011] Through the above design, this invention separates the graph topology and attribute data for storage, ensuring that the size of the resulting topology storage file does not exceed the maximum memory usage space. Thus, during topology traversal, the entire topology can be read into memory, avoiding the need for disk jumps along the chain structure required by traditional techniques, thereby improving topology access efficiency. Simultaneously, the fixed-length attribute record maintains the starting offset and length information corresponding to the storage of variable-length attribute data, and fixed-length attributes are stored sequentially by ID and loaded via memory mapping. Based on this, this invention allows direct access in O(1) time without deserialization, and variable-length attributes are accessed via offset indexes, avoiding the problem of multi-layered structure parsing. Therefore, this invention significantly improves the access performance of graph data and the system's concurrency capabilities, making it highly suitable for large-scale applications and promotion.

[0012] In one possible design, the graph topology information includes edge topology information and node topology information; The graph topology information is stored to obtain a graph topology file, including: A first structure array is constructed, and the edge topology information is stored using the first structure array to obtain an edge topology structure array; A node array is constructed, and the node topology information is stored in the node array to obtain a node topology array; The graph topology structure file is generated using the edge topology structure array and the node topology array.

[0013] In one possible design, the edge topology information includes: the ID of each edge in the RAG knowledge graph and the IDs of the two nodes connected to each edge. The first structure array has multiple consecutive subscripts, the number of subscripts is the same as the number of edges, and each subscript corresponds to an element structure in the first structure array. Specifically, the first structure array is used to store edge topology information, resulting in an edge topology structure array, including: Map and associate the ID of each edge with each index of the first structure array, wherein the ID of each edge is mapped and associated with a single index. For any edge, store the IDs of the two nodes connected by the edge into the target structure in the first structure array, so as to obtain the edge topology structure array after iterating through all edges in the RAG knowledge graph. The target structure is the element structure corresponding to the index of the ID mapping associated with the edge.

[0014] In one possible design, the node topology information includes: the ID of each node in the RAG knowledge graph, as well as the ID of the outgoing edge and the ID of the incoming edge of each node. The node array includes a first array and a second array. Both the first array and the second array have multiple consecutive subscripts. The number of subscripts in the first array and the second array is equal to the number of nodes. Each subscript in the first array corresponds to an outgoing edge array, and each subscript in the second array corresponds to an incoming edge array. Specifically, the node topology information is stored using the node array to obtain a node topology array, which includes: Map each node's ID to each index in the first and second arrays. Each node's ID is mapped to an index, and the index mapped to the ID of any node is the same in both the first and second arrays. For any node, store the ID of the outgoing edge of the node in the first specified array in the first array, and store the ID of the incoming edge of the node in the second specified array in the second array. After polling all nodes, obtain the node outgoing edge topology array and the node incoming edge topology array respectively, so as to form a node topology array using the node outgoing edge topology array and the node incoming edge topology array. Wherein, the first specified array is the outgoing edge array corresponding to the index associated with the ID mapping of any node in the first array, and the second specified array is the incoming edge array corresponding to the index associated with the ID mapping of any node in the second array.

[0015] In one possible design, the variable-length attribute data of each node is stored in a node variable-length attribute file to obtain the first starting storage location and first storage length of the variable-length attribute data of each node, including: Generate variable-length attribute records for each node based on the variable-length attribute data of each node; According to the ID order of each node, the variable length attribute records of each node are stored in the node variable length attribute file, so that the starting storage position and storage length of the variable length attribute records of each node in the node variable length attribute file are used as the first starting storage position and first storage length of the variable length attribute data of each node.

[0016] In one possible design, based on the variable-length attribute data of each node, variable-length attribute records for each node are generated, including: For any node, based on each data field in the variable-length attribute data of that node, construct a second structure array corresponding to each data field; The metadata record segment is composed of the second structure array of each data field; Use the various data fields to form variable-length attribute record segments; By concatenating the metadata record segment and the variable-length attribute record segment, the variable-length attribute record of any node is obtained.

[0017] In one possible design, based on the data fields in the variable-length attribute data of any node, a structure array corresponding to each data field is constructed, including: For any data field, the local starting offset of the data field is determined based on the field length of the preceding data field in the variable-length attribute data of the node. Using the local starting offset of any data field and the length of any data field, a second structure array corresponding to any data field is constructed.

[0018] Secondly, an efficient topology and attribute storage device for RAG knowledge graphs is provided, including: The parsing unit is used to parse the RAG knowledge graph to obtain the graph topology information, as well as the attribute information of each node and edge in the RAG knowledge graph; A topology storage unit is used to store the graph topology information to obtain a graph topology file; The partitioning unit is used to divide the attribute information of each node and edge into fixed-length attribute data and variable-length attribute data. The attribute storage unit is used to store the variable-length attribute data of each node and the variable-length attribute data of each edge into the node variable-length attribute file and the edge variable-length attribute file respectively, so as to obtain the first starting storage position and the first storage length of the variable-length attribute data of each node, and the second starting storage position and the second storage length of the variable-length attribute data of each edge respectively. The attribute storage unit is used to generate a fixed-length structure for each node based on the ID, first storage length, first starting storage position and fixed-length attribute data of each node, and to generate a fixed-length structure for each edge based on the ID, second storage length, second starting storage position and fixed-length attribute data of each edge. The attribute storage unit is also used to store the fixed-length structure of each node into the node fixed-length attribute file according to the ID order of each node, and to store the fixed-length structure of each edge into the edge fixed-length attribute file according to the ID order of each edge. After storage, the node fixed-length attribute file and the edge fixed-length attribute file are mapped into the process virtual memory through the mmap mechanism, so as to complete the storage of the topology and attribute information of the RAG knowledge graph after mapping.

[0019] Thirdly, another efficient topology and attribute storage device for RAG knowledge graphs is provided. Taking an electronic device as an example, it includes a memory, a processor, and a transceiver that are connected in sequence. The memory is used to store computer programs, the transceiver is used to send and receive messages, and the processor is used to read the computer programs and execute the efficient topology and attribute storage method for RAG knowledge graphs as described in the first aspect or any possible design of the first aspect.

[0020] Fourthly, a storage medium is provided, on which instructions are stored, which, when executed on a computer, perform the efficient topology and attribute storage method for RAG knowledge graphs as described in the first aspect or any possible design of the first aspect.

[0021] Fifthly, a computer program product containing instructions is provided, which, when executed on a computer, causes the computer to perform an efficient topology and attribute storage method for RAG knowledge graphs as described in the first aspect or any possible design of the first aspect.

[0022] Compared with the prior art, the present invention has the following advantages and beneficial effects: (1) The topology and attribute separation design of graph data structure can read the entire topology structure into memory, thereby avoiding the problem of traditional technology requiring disk jumps along the chain structure, thus improving the efficiency of topology access.

[0023] (2) Efficient attribute access: Fixed-length attributes are stored sequentially and loaded through memory mapping, and can be accessed directly in O(1) time without deserialization; at the same time, variable-length attributes are accessed through offset index, avoiding the problem of parsing multi-level structures.

[0024] (3) Cache-friendly and fast startup: Sequential layout improves cache hit rate and mmap mapping reduces loading overhead.

[0025] (4) By decoupling the topology and attribute data, using the attribute storage mechanism of fixed-length attributes to maintain variable-length attributes, and introducing the memory mapping mechanism, this invention can significantly improve the access performance of graph data and the concurrency capability of the system while maintaining the simplicity of the storage structure. Therefore, this invention is particularly suitable for graph data scenarios with stable structure and read-intensive nature, such as knowledge graphs and social network relationship graphs, and effectively solves the performance bottleneck problem of traditional chain structure in large graph attribute query. Attached Figure Description

[0026] To more clearly illustrate the technical solutions of the exemplary embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly described below. It should be understood that the following drawings only show some embodiments of the present invention and should not be considered as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort. In the drawings: Figure 1 A flowchart illustrating the steps of an efficient topology and attribute storage method for RAG knowledge graphs provided in an embodiment of the present invention; Figure 2 This is a schematic diagram illustrating the storage of edge topology information provided in an embodiment of the present invention; Figure 3 This is a schematic diagram illustrating the storage of outgoing edge IDs in the point topology information provided in an embodiment of the present invention. Figure 4 This is a schematic diagram of the structure of a variable-length attribute record provided in an embodiment of the present invention; Figure 5 A schematic diagram of the structure of an efficient topology and attribute storage device for RAG knowledge graphs provided in an embodiment of the present invention; Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to embodiments and accompanying drawings. The illustrative embodiments and descriptions of this invention are for explanation only and are not intended to limit the invention. It should be understood that although terms such as "first," "second," etc., may be used herein to describe various units, these units should not be limited by these terms. These terms are only used to distinguish one unit from another. For example, a first unit may be referred to as a second unit, and similarly, a second unit may be referred to as a first unit, without departing from the scope of the exemplary embodiments of this invention.

[0028] Example: See Figure 1 As shown, the efficient topology and attribute storage method for RAG knowledge graphs provided in this embodiment, through the decoupling of topology structure and attribute data, the attribute storage mechanism of fixed-length attributes maintaining variable-length attributes, and the introduction of a memory mapping mechanism, can significantly improve the access performance of graph data and the system concurrency capability while maintaining a simple storage structure. Therefore, this method is very suitable for large-scale application and promotion. For example, this method can be run on the graph analysis system side, but is not limited to it. It is understood that the aforementioned execution entity does not constitute a limitation on the embodiments of this application. Accordingly, the running steps of this method can be, but are not limited to, the steps S1 to S6 below.

[0029] S1. Parse the RAG knowledge graph to obtain graph topology information and attribute information of each node and edge in the RAG knowledge graph; in specific implementation, the RAG knowledge graph is a directed knowledge graph. The RAG knowledge graph to be analyzed can be obtained according to actual use, and it can be parsed to obtain the corresponding graph topology information and graph attribute information.

[0030] In this embodiment, the graph topology information may include, but is not limited to, edge topology information and node topology information. The edge topology information includes the ID of each edge in the RAG knowledge graph and the IDs of the two nodes connected to each edge. The node topology information includes the ID of each node in the RAG knowledge graph, as well as the IDs of the outgoing edges and incoming edges of each node. Specifically, the outgoing edge of any node refers to the edge from that node to other nodes, while the incoming edge of any node is the edge that points to that node.

[0031] Meanwhile, the attribute information of the example graph may include, but is not limited to, the attribute information of each node and edge in the RAG knowledge graph. For example, the attribute of a node may correspond to a person's name, age, mobile phone number, etc., and the aforementioned name, age, and mobile phone number are the attribute information of that node. Of course, the content of the attribute information of other nodes and edges will not be elaborated one by one.

[0032] After parsing the RAG knowledge graph and obtaining the corresponding graph topology and attribute information, the topology and attribute information can be stored. Specifically, this embodiment provides a storage method that separates graph topology and attribute data, arranges IDs sequentially, and uses memory mapping, thereby solving problems such as mixed storage of topology and attributes, chained access, and high overhead from multiple jumps in existing graph databases.

[0033] The storage procedure for the topology structure is first disclosed, as shown in step S2 below.

[0034] S2. Store the graph topology information to obtain a graph topology file; in specific implementation, this embodiment uses array mapping to store the graph topology information, and the process is shown in steps S21 to S23 below.

[0035] S21. Construct a first structure array and use the first structure array to store the edge topology information to obtain an edge topology structure array. In this embodiment, for example, the first structure array may have multiple consecutive subscripts (such as 0, 1, 2, 3, ..., N, where N represents the number of edges), the number of subscripts is the same as the number of edges, and each subscript corresponds to an element structure in the first structure array. Based on this, the process of using the first structure array to store the edge topology is the mapping between the edge ID and the array subscript, and the storage of the two endpoints of the edge. The process may be, but is not limited to, the steps S21a and S21b below.

[0036] S21a. Map and associate the ID of each edge with each index of the first structure array, wherein each edge ID is mapped to an index. In specific applications, the ID of each edge in the RAG knowledge graph is mapped one-to-one with the array index. For example, if the edge ID is 100, the corresponding index is 0; if the edge ID is 101, the corresponding index is 1, and so on. In this way, after mapping the edge IDs to the indices of the first structure array in the aforementioned manner, the IDs of the two endpoints of each edge can be stored, as shown in step S21b below.

[0037] S21b. For any edge, store the IDs of the two nodes connected by the edge into the target structure in the first structure array, so as to obtain the edge topology structure array after polling all edges in the RAG knowledge graph, wherein the target structure is the element structure corresponding to the index associated with the ID mapping of the edge.

[0038] In this embodiment, the IDs of the two nodes connected by each edge (i.e., the two endpoints of the edge) are stored in the element structure corresponding to the index mapped to the edge ID, so as to realize the associated storage of the edge ID and its corresponding two endpoint IDs; specifically, the following example illustrates this: Assume the first structure array is named: vector <edgeinfo>edge_topo, where EdgeInfo is an element structure that includes two fields: src_nodeid and dst_nodeid. These two fields are used to record the IDs of the two endpoints of the edge.

[0039] Also see Figure 2 As shown, for an edge with ID 100, it connects to two nodes, A and B, where node A has ID 100 and node B has ID 101 (node ​​and edge IDs are independent and can be repeated). Assuming the edge with ID 100 has an array index of 0, then the IDs of nodes A and B are stored in the structure corresponding to index 0. See [link to relevant documentation]. Figure 2 As shown, Figure 2 The "src_nodeid:100,dst_nodeid:101" in the text indicates the element structure corresponding to index 0, which records the IDs of node A and node B.

[0040] Similarly, for the edge with ID 101, the two nodes it connects are nodes A and C; where the array index mapped to the edge with ID 101 is 1, then the element structure at index 1 stores the IDs of nodes A and C, that is... Figure 2 In the example, src_nodeid:100, dst_nodeid:102; of course, for the other edges and their corresponding two endpoints, the storage procedure is the same as the example above, and will not be repeated here.

[0041] Therefore, through the aforementioned steps S21a and S21b, the storage of the edge topology structure can be completed, and an array of edge topology structures can be obtained; then, the storage of node topology information can be performed, as shown in step S22 below.

[0042] S22. Construct a node array and use the node array to store the node topology information to obtain a node topology array; in specific implementations, the node array may, but is not limited to, include a first array (which may be named vector). <vector<uint32_t> >node_out_topo) and the second array (which can be named vector) <vector<uint32_t> >node_in_topo), where the first array and the second array both have multiple consecutive indices, the number of indices in the first array and the second array are equal to the number of nodes, and each index in the first array corresponds to an outgoing edge array, and each index in the second array corresponds to an incoming edge array; thus, in this embodiment, the first array is used to record the information of the node and its corresponding outgoing edge, and the second array is used to record the information of the node and its corresponding incoming edge; optionally, the aforementioned recording process can be, but is not limited to, as shown in steps S22a and S22b below.

[0043] S22a. Map and associate the IDs of each node with the indices of the first array and the second array, wherein each node ID is mapped to an index, and the index mapped to any node ID in the first array and the second array is the same. In this embodiment, the mapping process of node IDs with the indices of the first array and the second array is the same as the mapping process of edge IDs with the indices of the first structure array, which is a one-to-one mapping, and will not be described in detail here.

[0044] After completing the association mapping between node IDs and the two array indices, the IDs of the outgoing edges and incoming edges of the nodes can be recorded, as shown in step S22b below.

[0045] S22b. For any node, store the ID of the outgoing edge of the node in the first designated array of the first array, and store the ID of the incoming edge of the node in the second designated array of the second array. After polling all nodes, obtain the node outgoing edge topology array and the node incoming edge topology array respectively. In this embodiment, the first designated array is the outgoing edge array corresponding to the index of the ID mapping associated with the node in the first array, and the second designated array is the incoming edge array corresponding to the index of the ID mapping associated with the node in the second array.

[0046] Optionally, the following example illustrates the aforementioned steps S22a and S22b: See Figure 3 As shown, for node A, its ID is 100, and its corresponding outgoing edges are edges with IDs 101 and 100. Assume that the indices of the first and second arrays mapped to the ID of node A are both 0. The first array is used to record the IDs of the outgoing edges of node A. See [link to example]. Figure 3 As shown, the outgoing edge array with index 0 in the first array is {100, 101}.

[0047] Similarly, for nodes B and C, assuming that their mapping indices are 1 and 2 respectively, and since nodes B and C do not have any outgoing edges, the outgoing edge arrays corresponding to indices 1 and 2 in the first array are empty sets. Of course, when the number of nodes is greater than the previous example, the recording method of the corresponding outgoing edge IDs is the same as the previous example, and will not be repeated here.

[0048] Furthermore, for the ID record of the incoming edge of node A, the index mapped to its ID in the second array is still 0. However, since node A has no incoming edges, the incoming edge array corresponding to index 0 in the second array is an empty set. For node B, the index mapped to its ID in the second array is still 1, and the ID of the incoming edge of node B is 100. Therefore, the incoming edge array corresponding to index 1 in the second array is {100}. Similarly, for node C, the index mapped to its ID in the second array is 2, and the ID of its incoming edge is 101. Therefore, the incoming edge array corresponding to index 2 in the second array is {101}.

[0049] Therefore, by going through the aforementioned steps S22a and S22b, the node topology information can be recorded, thereby obtaining the node outgoing edge topology array and the node incoming edge topology array. Then, the two can be used to form the node topology array. Finally, the aforementioned edge topology array can be combined to form the graph topology structure file, as shown in step S23 below.

[0050] S23. Using the edge topology structure array and the node topology array, generate the graph topology structure file; in this embodiment, the graph topology structure file includes the aforementioned edge topology structure array, node outgoing edge topology array, and node incoming edge topology array; thus, after obtaining the graph topology structure file, it can be stored on the disk; in addition, in this embodiment, the storage of the example graph topology structure information is all performed in memory, that is, the storage procedure is executed in memory and the storage file is written to the disk.

[0051] Thus, through the aforementioned steps S21 to S23, the storage of graph topology information can be completed; then, the attribute information of each node and edge can be stored; in this embodiment, an attribute storage method is provided that uses fixed-length attributes to maintain the storage start offset and storage length of variable-length attribute data, and a memory mapping loading method is introduced at the same time to improve attribute access efficiency and reduce the number of attribute access jumps.

[0052] Optionally, the attribute stored procedure may be, but is not limited to, the steps S3 to S6 below.

[0053] S3. Divide the attribute information of each node and edge into fixed-length attribute data and variable-length attribute data. In this embodiment, the attributes are first divided into fixed-length attributes and variable-length attributes, and then the variable-length attributes are stored first and the fixed-length attributes are stored later. For example, the fixed-length attributes are integer or floating-point data, while the variable-length attributes are strings.

[0054] Thus, after the attribute information of nodes and edges is divided, variable-length attributes can be stored, as shown in step S4 below.

[0055] S4. Store the variable length attribute data of each node and the variable length attribute data of each edge into the node variable length attribute file and the edge variable length attribute file respectively, so as to obtain the first starting storage position and the first storage length of the variable length attribute data of each node, and the second starting storage position and the second storage length of the variable length attribute data of each edge respectively.

[0056] In practical applications, taking the variable-length attribute of a node as an example, the specific explanation is as follows: its storage procedure can be, but is not limited to, the steps S41 and S42 shown below.

[0057] S41. Generate variable-length attribute records for each node based on the variable-length attribute data of each node; in this embodiment, for example, but not limited to, the following steps S41a to S41c can be used to generate variable-length attribute records for any node.

[0058] S41a. For any node, based on each data field in the variable-length attribute data of the node, construct a second structure array corresponding to each data field; in specific applications, for any data field, the local starting offset of the data field can be determined based on, but is not limited to, the length of the preceding data field in the variable-length attribute data of the node; then, the local starting offset and the length of the data field are used to construct the second structure array corresponding to the data field.

[0059] In this embodiment, the local start offset of any data field is the start partial offset of the previous data field of the any data field + the length of the previous data field; and when the any data field is the first field, its local start offset is 0.

[0060] Specifically, an example is used to elaborate: Suppose there is an attribute information of a node, which is: {"id": 152, "label": "phone", "prop":{"name": "Cao xx", "age": 35, "sex": 1, "phone_num": "189403202xx"}}; among them, "Cao xx" and "189403202xx" are variable-length attribute data, and "35", "1" are fixed-length attribute data; then, the data fields in the variable-length attribute data are "Cao xx" and "189403202xx".

[0061] Specifically, for "Cao xx", it is the first data field in the variable-length attribute data. Therefore, its local start offset is 0, that is, start_offset = 0; at the same time, its byte length is 6, so lens = 6. Thus, the second structure array corresponding to "Cao xx" is: start_offset = 0; lens = 6.

[0062] Similarly, for the data field "189403202xx", it is after "Cao xx". Therefore, its corresponding local start offset is the length of "Cao xx" + the local start offset corresponding to "Cao xx", that is, its start_offset = 0 + 6 = 6, and at the same time, its length is lens = 11; thus, the second structure array corresponding to "189403202xx" is: start_offset = 6; lens = 11; for its schematic diagram, see Figure 4 as shown.

[0063] After obtaining the second structure arrays corresponding to each data field, the second structure arrays of each data field can be used to form a metadata record segment, and the process is as shown in the following step S41b.

[0064] S41b. Use the second structure arrays of each data field to form a metadata record segment.

[0065] After obtaining the metadata record segment, each data field can be used to form a variable-length attribute record segment, and the process is as shown in the following step S41c.

[0066] S41c. Use each data field to form a variable-length attribute record segment; in this embodiment, the variable-length attribute record segment is "Cao xx" + "189403202xx"; thus, by splicing the variable-length attribute record segment and the metadata record segment, a variable-length attribute record can be obtained, and the process is as shown in the following step S41d. S41d. Splice the metadata record segment and the variable-length attribute record segment to obtain the variable-length attribute record of any one of the nodes; in this embodiment, the final variable-length attribute record can be seen in Figure 4 as shown; at this time, the length of the entire variable-length attribute record segment is the sum of the lengths of two second structure arrays and the lengths of two data fields; where it is assumed that start_offse and lens each occupy four bytes, then the length of the foregoing variable-length attribute record segment is 8×2 + 17 = 33.

[0067] Thus, after obtaining the variable-length attribute records of each node through the foregoing steps S41a to S41d, they can be stored in the node variable-length attribute file, and the process is as shown in the following step S42.

[0068] S42. Store the variable-length attribute records of each node in the node variable-length attribute file in the order of the IDs of each node, and use the starting storage position and storage length of each node's variable-length attribute record in the node variable-length attribute file as the first starting storage position and the first storage length of each node's variable-length attribute data; in this embodiment, it is assumed that the ID of the node is 152, then the variable-length attribute record of this node is stored after the variable-length attribute record of the node with ID 151; at this time, after storage, the storage starting offset (starting storage position) and storage length (i.e., the length of the variable-length attribute record) of the variable-length attribute record of the node with ID 152 can be obtained. Based on this, the first storage position and the first starting offset position of the node with ID 152 can be obtained; of course, the determination process of the first storage position and the first starting offset position of each of the remaining nodes is also the same, and will not be elaborated here.

[0069] In addition, for the variable-length attribute data of each edge, its storage process is the same as that of the variable-length attribute data of the foregoing nodes, and will not be elaborated here.

[0070] Thus, after obtaining the storage lengths and starting storage positions of the variable-length attribute data of each node and edge in their respective variable-length attribute files through the foregoing steps S41 to S42, the fixed-length attribute can be stored, and the process is as shown in the following step S5.

[0071] S5. Based on the ID, first storage length, first starting storage position and fixed-length attribute data of each node, generate a fixed-length structure for each node, and based on the ID, second storage length, second starting storage position and fixed-length attribute data of each edge, generate a fixed-length structure for each edge.

[0072] In practical applications, this is equivalent to recording the ID, first storage length, first starting storage location, and fixed-length attribute data of each node together, thereby forming a fixed-length structure for each node.

[0073] As illustrated in the previous example, the fixed-length attribute data includes "35" and "1"; meanwhile, the first storage length of the variable-length attribute data of the node is 33, and assuming its first starting storage position is 456; then, the fixed-length structure of this node can be represented as: struct node_record{uint32_t id; / / 152; uint32_t age; / / 35; uint32_tsex; / / 1; uint32_t prop_start_offset; / / 456; uint32_t prop_lens; / / 33}.

[0074] Of course, the same process applies to generating fixed-length structures on each side, which will not be elaborated here.

[0075] Thus, after obtaining the fixed-length structures of each node and edge, the fixed-length structures of each node and edge can be stored in order of ID, as shown in step S6 below.

[0076] S6. Store the fixed-length structure of each node in the node fixed-length attribute file according to the ID order of each node, and store the fixed-length structure of each edge in the edge fixed-length attribute file according to the ID order of each edge. After storage, map the node fixed-length attribute file and the edge fixed-length attribute file into the process virtual memory through the mmap mechanism, so as to complete the storage of the topology and attribute information of the RAG knowledge graph after mapping.

[0077] In this embodiment, assuming a node's fixed-length structure occupies 20 bytes, and its starting offset in the node's fixed-length attribute file is 20 × ID (ID is 152) = 3040, the storage location of the corresponding fixed-length structure can be obtained by multiplying the ID by the size of the fixed-length structure. Furthermore, as described above, the fixed-length structure contains the storage length and starting storage location of variable-length attribute data. Therefore, after obtaining the fixed-length structure located in the node's fixed-length attribute file, the variable-length attribute data can be read based on the starting offset and storage length of the variable-length record within the fixed-length structure. Based on this, the design avoids chained structure traversal, simplifies the access process, and only requires a single jump to obtain the complete variable-length attribute data, thereby improving data decoding efficiency.

[0078] Meanwhile, the fixed-length attribute file is mapped to the process virtual memory through the mmap mechanism, allowing visitors to directly calculate the offset and read the corresponding record by node ID, which can achieve efficient access with O(1) time complexity without manual deserialization or intermediate data structure parsing; and the attribute data is arranged in ID order, which makes the data have good spatial locality during the access process, which is convenient for CPU prefetching and cache hit; at the same time, since there is no need for frequent disk jumps and intermediate index layer parsing, the I / O latency in the graph traversal and attribute query process can be significantly reduced.

[0079] Finally, after mapping the fixed-length attribute files, they can be stored on disk along with the graph topology file. Upon the next system startup, the graph topology file can be read into memory, enabling efficient topology access. Similarly, when accessing attribute data, all fixed-length attribute files can be loaded via memory mapping (mmap) to achieve zero-copy access. Then, the complete variable-length attribute data can be quickly derived using the starting offset and length information of the variable-length records maintained in the fixed-length structure within the variable-length attribute file. Based on this, graph data access can be achieved quickly.

[0080] Therefore, through the efficient topology and attribute storage method for RAG knowledge graphs described in detail in steps S1 to S6 above, this invention, by decoupling the topology structure and attribute data, the attribute storage mechanism of fixed-length attributes maintaining variable-length attributes, and the introduction of a memory mapping mechanism, can significantly improve the access performance of graph data and the system concurrency capability while maintaining a simple storage structure. Therefore, it is very suitable for large-scale application and promotion.

[0081] like Figure 5 As shown, the second aspect of this embodiment provides a hardware device for implementing the efficient topology and attribute storage method for RAG knowledge graphs described in the first aspect of the embodiment, comprising: The parsing unit is used to parse the RAG knowledge graph to obtain the graph's topological structure information, as well as the attribute information of each node and edge in the RAG knowledge graph.

[0082] A topology storage unit is used to store the graph topology information to obtain a graph topology file.

[0083] The partitioning unit is used to divide the attribute information of each node and edge into fixed-length attribute data and variable-length attribute data.

[0084] The attribute storage unit is used to store the variable-length attribute data of each node and the variable-length attribute data of each edge into the node variable-length attribute file and the edge variable-length attribute file, respectively, so as to obtain the first starting storage position and the first storage length of the variable-length attribute data of each node, and the second starting storage position and the second storage length of the variable-length attribute data of each edge.

[0085] The attribute storage unit is used to generate a fixed-length structure for each node based on the ID, first storage length, first starting storage position and fixed-length attribute data of each node, and to generate a fixed-length structure for each edge based on the ID, second storage length, second starting storage position and fixed-length attribute data of each edge.

[0086] The attribute storage unit is also used to store the fixed-length structure of each node into the node fixed-length attribute file according to the ID order of each node, and to store the fixed-length structure of each edge into the edge fixed-length attribute file according to the ID order of each edge. After storage, the node fixed-length attribute file and the edge fixed-length attribute file are mapped into the process virtual memory through the mmap mechanism, so as to complete the storage of the topology and attribute information of the RAG knowledge graph after mapping.

[0087] The working process, working details and technical effects of the device provided in this embodiment can be found in the first aspect of the embodiment, and will not be repeated here.

[0088] like Figure 6 As shown, the third aspect of this embodiment provides another efficient topology and attribute storage device for RAG knowledge graphs. Taking an electronic device as an example, it includes: a memory, a processor, and a transceiver that are connected in sequence. The memory is used to store computer programs, the transceiver is used to send and receive messages, and the processor is used to read the computer programs and execute an efficient topology and attribute storage method for RAG knowledge graphs as described in the first aspect of the embodiment.

[0089] For specific examples, the memory may include, but is not limited to, random access memory (RAM), read-only memory (ROM), flash memory, first-in-first-out (FIFO) memory, and / or first-in-last-out (FILO) memory, etc.; specifically, the processor may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor may be implemented using at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), PLA (Programmable Logic Array). The processor may also include a main processor and a coprocessor. The main processor, also known as the CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state.

[0090] In some embodiments, the processor may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. For example, the processor may not be limited to microprocessors of the STM32F105 series, reduced instruction set computer (RISC) microprocessors, x86 architecture processors, or processors with integrated neural network processing units (NPUs). The transceiver may be, but is not limited to, a Wi-Fi transceiver, a Bluetooth transceiver, a General Packet Radio Service (GPRS) transceiver, a ZigBee (a low-power LAN protocol based on the IEEE 802.15.4 standard) transceiver, a 3G transceiver, a 4G transceiver, and / or a 5G transceiver. Furthermore, the device may also include, but is not limited to, a power module, a display screen, and other necessary components.

[0091] The working process, working details and technical effects of the electronic device provided in this embodiment can be found in the first aspect of the embodiment, and will not be repeated here.

[0092] The fourth aspect of this embodiment provides a storage medium that stores instructions containing the efficient topology and attribute storage method for RAG knowledge graphs as described in the first aspect of this embodiment. That is, the storage medium stores instructions that, when executed on a computer, perform the efficient topology and attribute storage method for RAG knowledge graphs as described in the first aspect of this embodiment.

[0093] The storage medium refers to a carrier for storing data, which may include, but is not limited to, floppy disks, optical disks, hard disks, flash memory, USB flash drives, and / or memory sticks. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices.

[0094] The working process, working details and technical effects of the storage medium provided in this embodiment can be found in the first aspect of the embodiment, and will not be repeated here.

[0095] The fifth aspect of this embodiment provides a computer program product containing instructions that, when executed on a computer, cause the computer to perform an efficient topology and attribute storage method for RAG knowledge graphs as described in the first aspect of this embodiment, wherein the computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device.

[0096] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.< / edgeinfo>

Claims

1. An efficient topology and attribute storage method for RAG knowledge graphs, characterized in that, include: Parse the RAG knowledge graph to obtain the graph topology information, as well as the attribute information of each node and edge in the RAG knowledge graph; The graph topology information is stored to obtain a graph topology file; The attribute information of each node and edge is divided into fixed-length attribute data and variable-length attribute data; The variable length attribute data of each node and the variable length attribute data of each edge are stored in the node variable length attribute file and the edge variable length attribute file respectively, so as to obtain the first starting storage position and the first storage length of the variable length attribute data of each node, and the second starting storage position and the second storage length of the variable length attribute data of each edge respectively. Based on the ID, first storage length, first starting storage position and fixed length attribute data of each node, a fixed length structure of each node is generated, and based on the ID, second storage length, second starting storage position and fixed length attribute data of each edge, a fixed length structure of each edge is generated. According to the ID order of each node, the fixed-length structure of each node is stored in the node fixed-length attribute file, and according to the ID order of each edge, the fixed-length structure of each edge is stored in the edge fixed-length attribute file. After storage, the node fixed-length attribute file and the edge fixed-length attribute file are mapped to the process virtual memory through the mmap mechanism, so as to complete the storage of the topology and attribute information of the RAG knowledge graph.

2. The method according to claim 1, characterized in that, The graph topology information includes edge topology information and node topology information; The graph topology information is stored to obtain a graph topology file, including: A first structure array is constructed, and the edge topology information is stored using the first structure array to obtain an edge topology structure array; A node array is constructed, and the node topology information is stored in the node array to obtain a node topology array; The graph topology structure file is generated using the edge topology structure array and the node topology array.

3. The method according to claim 2, characterized in that, The edge topology information includes: the ID of each edge in the RAG knowledge graph and the IDs of the two nodes connected to each edge. The first structure array has multiple consecutive subscripts, the number of subscripts is the same as the number of edges, and each subscript corresponds to an element structure in the first structure array. Specifically, the first structure array is used to store edge topology information, resulting in an edge topology structure array, including: Map and associate the ID of each edge with each index of the first structure array, wherein the ID of each edge is mapped and associated with a single index. For any edge, store the IDs of the two nodes connected by the edge into the target structure in the first structure array, so as to obtain the edge topology structure array after iterating through all edges in the RAG knowledge graph. The target structure is the element structure corresponding to the index of the ID mapping associated with the edge.

4. The method according to claim 2, characterized in that, The node topology information includes: the ID of each node in the RAG knowledge graph, as well as the ID of the outgoing edge and the ID of the incoming edge of each node. The node array includes a first array and a second array. Both the first array and the second array have multiple consecutive subscripts. The number of subscripts in the first array and the second array is equal to the number of nodes. Each subscript in the first array corresponds to an outgoing edge array, and each subscript in the second array corresponds to an incoming edge array. Specifically, the node topology information is stored using the node array to obtain a node topology array, which includes: Map each node's ID to each index in the first and second arrays. Each node's ID is mapped to an index, and the index mapped to the ID of any node is the same in both the first and second arrays. For any node, store the ID of the outgoing edge of the node in the first specified array in the first array, and store the ID of the incoming edge of the node in the second specified array in the second array. After polling all nodes, obtain the node outgoing edge topology array and the node incoming edge topology array respectively, so as to form a node topology array using the node outgoing edge topology array and the node incoming edge topology array. Wherein, the first specified array is the outgoing edge array corresponding to the index associated with the ID mapping of any node in the first array, and the second specified array is the incoming edge array corresponding to the index associated with the ID mapping of any node in the second array.

5. The method according to claim 1, characterized in that, The variable-length attribute data of each node is stored in the node variable-length attribute file to obtain the first starting storage location and the first storage length of the variable-length attribute data of each node, including: Generate variable-length attribute records for each node based on the variable-length attribute data of each node; According to the ID order of each node, the variable length attribute records of each node are stored in the node variable length attribute file, so that the starting storage position and storage length of the variable length attribute records of each node in the node variable length attribute file are used as the first starting storage position and first storage length of the variable length attribute data of each node.

6. The method according to claim 5, characterized in that, Based on the variable-length attribute data of each node, generate variable-length attribute records for each node, including: For any node, based on each data field in the variable-length attribute data of that node, construct a second structure array corresponding to each data field; The metadata record segment is composed of the second structure array of each data field; Use the various data fields to form variable-length attribute record segments; By concatenating the metadata record segment and the variable-length attribute record segment, the variable-length attribute record of any node is obtained.

7. The method according to claim 6, characterized in that, Based on the data fields in the variable-length attribute data of any node, construct a structure array corresponding to each data field, including: For any data field, the local starting offset of the data field is determined based on the field length of the preceding data field in the variable-length attribute data of the node. Using the local starting offset of any data field and the length of any data field, a second structure array corresponding to any data field is constructed.

8. A high-efficiency topology and attribute storage device for RAG knowledge graphs, characterized in that, include: The parsing unit is used to parse the RAG knowledge graph to obtain the graph topology information, as well as the attribute information of each node and edge in the RAG knowledge graph; A topology storage unit is used to store the graph topology information to obtain a graph topology file; The partitioning unit is used to divide the attribute information of each node and edge into fixed-length attribute data and variable-length attribute data. The attribute storage unit is used to store the variable-length attribute data of each node and the variable-length attribute data of each edge into the node variable-length attribute file and the edge variable-length attribute file respectively, so as to obtain the first starting storage position and the first storage length of the variable-length attribute data of each node, and the second starting storage position and the second storage length of the variable-length attribute data of each edge respectively. The attribute storage unit is used to generate a fixed-length structure for each node based on the ID, first storage length, first starting storage position and fixed-length attribute data of each node, and to generate a fixed-length structure for each edge based on the ID, second storage length, second starting storage position and fixed-length attribute data of each edge. The attribute storage unit is also used to store the fixed-length structure of each node into the node fixed-length attribute file according to the ID order of each node, and to store the fixed-length structure of each edge into the edge fixed-length attribute file according to the ID order of each edge. After storage, the node fixed-length attribute file and the edge fixed-length attribute file are mapped into the process virtual memory through the mmap mechanism, so as to complete the storage of the topology and attribute information of the RAG knowledge graph after mapping.

9. An electronic device, characterized in that, include: A memory, a processor, and a transceiver are sequentially connected in communication, wherein the memory is used to store computer programs, the transceiver is used to send and receive messages, and the processor is used to read the computer programs and execute the efficient topology and attribute storage method for RAG knowledge graphs as described in any one of claims 1 to 7.

10. A computer program product containing instructions, characterized in that, When the instructions are executed on the computer, the computer performs the efficient topology and attribute storage method for RAG knowledge graphs as described in any one of claims 1 to 7.

Images