Graph data processing method based on cache optimization

Abstract

The invention relates to the technical field of graph data processing, and provides a graph data processing method based on cache optimization, which comprises the following steps: constructing a cache optimization representation of graph data, and distributing a bit vector GT-vector with a fixed length of k for each vertex in a graph, each bit representing whether the vertex belongs to a pre-calculated maximum independent set; reserving adjacency list representation of the graph data, and storing the adjacency list representation as reference data in a memory; for an edge query request, reading GT-vectors of two vertexes from a CPU cache, and executing bit and operation; if the bit and the result are non-zero, judging that the edge is non-edge and immediately returning a first Boolean result indicating that the edge does not exist; and if the bit and result is zero, querying the adjacency list in the memory for verification, and returning a second Boolean result indicating that the edge exists or does not exist. According to the method and the device, the hybrid architecture combining the bit vector representation of cache optimization and the adjacency list is constructed, so that the magnitude order improvement of the graph data processing performance is realized on the premise of ensuring that the query result is completely accurate.

Description

Technical Field

[0001] This invention belongs to the field of graph data processing technology, and in particular relates to a graph data processing method based on cache optimization. Background Technology

[0002] Graph data representation and processing are key supporting technologies for modern large-scale data computing systems, and are widely used in core areas such as social network analysis, knowledge graph construction, financial risk control, and artificial intelligence. However, real-world graph data typically has extremely large scale (e.g., billions of vertices and trillions of edges), high sparsity, and irregular connectivity. These characteristics cause traditional graph processing methods to face severe performance bottlenecks under modern CPU architectures, especially due to limitations in cache access efficiency.

[0003] Currently, the mainstream methods for graph data representation and processing mainly fall into the following categories, but all of them have significant drawbacks: 1. Adjacency matrix: The graph is represented by a two-dimensional array of |V|×|V|, and edge queries can be completed in O(1) time; Drawback: The space complexity is O(|V|²), which requires a huge amount of storage space (TB level) for large-scale graphs (such as those with tens of millions of vertices), making it completely impractical; 2. Adjacency List: Stores a list of neighboring vertices for each vertex, with a space complexity of O(|V|+|E|), and is currently the most mainstream sparse graph representation method; Drawbacks: Because the neighbor list is stored in a scattered manner in memory, a large number of irregular memory accesses are generated during the access process, resulting in an extremely high L3 cache miss rate; in sparse graphs, more than 99.99% of edge queries are to verify "non-edges"; using the adjacency list to verify non-edges requires traversing or searching the neighbor list, which triggers 2-4 L3 cache misses on average per query, consuming more than 70% of the query time; the neighbor list for height-numbered nodes spans multiple cache lines, while for low-degree nodes, most of the cache line space is wasted; 3. Existing optimization techniques: Locality optimization techniques: Improve spatial locality of access through graph partitioning or vertex reordering (such as Gorder, METIS). However, this method cannot change the fundamental access pattern that adjacency lists require traversing linked lists. For non-edge queries, which account for a very large proportion, its caching performance improvement is limited. Graph compression / digest techniques: Reduce data volume by using lossy or lossless compression algorithms. However, this method often sacrifices the accuracy of query results, which may introduce false positives or require additional decompression computation overhead. Pre-computed indexing techniques: Pre-build auxiliary index structures for specific types of queries (such as reachability queries). The main problem with this technique is its poor generality. The index itself will bring huge storage overhead and cannot efficiently support basic edge existence queries.

[0004] Therefore, existing technologies lack a graph data processing method that can simultaneously meet the three major requirements of efficient caching, accurate results, and strong versatility. Summary of the Invention

[0005] In view of the above-mentioned deficiencies of the prior art, this invention proposes a graph data processing method based on cache optimization. By pre-computing the maximum independent set, the majority of non-edge relations in the graph are encoded into compact bit vectors, which can reside in the CPU cache. During a query, non-edge determination is first completed in the cache through a single-cycle bit operation. Only for a few queries that cannot be determined, the method is rolled back to the exact adjacency list in memory for verification. The technical solution designed in this invention includes the following steps: To construct a cache-optimized representation of graph data, each vertex in the graph is assigned a bit vector GT-vector of fixed length k, where each bit indicates whether the vertex belongs to a pre-computed maximum independent set; The adjacency list representation of the graph data is preserved and stored in memory as the base data; For edge query requests, read the GT-vectors of the two vertices from the CPU cache and perform a bitwise AND operation; If the bit AND result is non-zero, it is determined to be a non-edge and the first Boolean result indicating that the edge does not exist is immediately returned; If the bitwise AND result is zero, the adjacency list in memory is queried for verification, and a second Boolean result indicating whether the edge exists or not is returned.

[0006] Preferably, the cache-optimized representation of the constructed graph data includes: Implemented using the GTWICE algorithm, the GT-vector of each vertex is initialized to all zeros, and a counter is initialized to record the number of times each vertex is selected into the maximum independent set, and k iterations are performed: S1: Sort all vertices in ascending order according to the value of the counter to generate a vertex sequence; S2: Based on the vertex sequence, use a greedy algorithm to calculate the i-th largest independent set; S3: Set the i-th position of the GT-vector corresponding to the vertex belonging to the largest independent set to 1, and increment the counter.

[0007] Preferably, the time complexity of the GTWICE algorithm is O(k(|V|+|E|)), where |V| is the number of vertices, |E| is the number of edges, and k is the predefined bit vector length.

[0008] Preferably, the length of the predefined bit vector is a multiple of 64 bits.

[0009] Preferably, the bitwise AND operation is completed within a single CPU cycle.

[0010] Preferably, the adjacency list in memory is queried using a binary search algorithm.

[0011] Preferably, the method supports multi-core parallel processing of multiple edge query requests without dependencies.

[0012] Beneficial effects: 1. This invention reduces the response time of most non-edge queries from tens of nanoseconds in traditional adjacency lists to a single CPU cycle by caching GT-vectors bit vectors, thereby reducing query latency and breaking through the limitation of memory walls on graph processing performance. 2. This invention adopts a hybrid architecture that combines GT-vectors fast filtering with adjacency list precise verification. While maintaining high-speed query, it completely eliminates the risk of false positives and false negatives, providing a reliable correctness guarantee for graph data analysis tasks. 3. The design of this invention based on bit vectors is naturally adapted to the SIMD instruction set and multi-core parallel architecture of modern CPUs. Its linear space complexity ensures that ultra-large-scale graph data can reside in the high-speed cache and can be seamlessly integrated into existing graph processing methods as a general acceleration layer. Attached Figure Description

[0013] Figure 1 This is a flowchart illustrating a preferred embodiment of the present invention; Figure 2 This is a schematic diagram of a social network according to a preferred embodiment of the present invention; Figure 3 This is a schematic diagram of GT-vectors according to a preferred embodiment of the present invention. Detailed Implementation

[0014] The embodiments of the present invention will be described in detail below. The embodiments described below are implemented based on the technical solution of the present invention, and detailed implementation methods and specific operation processes are given. However, the protection scope of the present invention is not limited to the embodiments described below.

[0015] This invention designs a graph data processing method based on cache optimization. The technical solution includes the following steps, such as... Figure 1 As shown, it specifically includes: To construct a cache-optimized representation of graph data, each vertex in the graph is assigned a bit vector GT-vector of fixed length k, where each bit indicates whether the vertex belongs to a pre-computed maximum independent set; The adjacency list representation of the graph data is preserved and stored in memory as the base data; For edge query requests, read the GT-vectors of the two vertices from the CPU cache and perform a bitwise AND operation; If the bit AND result is non-zero, it is determined to be a non-edge and the first Boolean result indicating that the edge does not exist is immediately returned; If the bitwise AND result is zero, the adjacency list in memory is queried for verification, and a second Boolean result indicating whether the edge exists or not is returned.

[0016] Specifically, a value of 1 in the i-th bit vector indicates that vertex v belongs to the pre-computed i-th Maximum Independent Set (MIS), while a value of 0 indicates that it does not. The total space cost of this representation is k|V| bits, or O(k|V|), which is linearly related only to the number of vertices and independent of edge density. For example, for a graph containing 10 million vertices, only about 80MB of storage is required, which can be fully loaded into the L3 cache of a modern CPU.

[0017] Furthermore, the adjacency list, serving as a "fact source," retains the complete traditional adjacency list structure for precise verification of queries that GT-vectors cannot cover, ensuring the absolute correctness of the results. This invention employs a two-layer hybrid architecture to coordinate the two data representations mentioned above, enabling efficient hybrid query processing.

[0018] Furthermore, according to the definition of independent sets, if the bitwise AND result of the GT-vectors of two vertices is non-zero, then they share at least one independent set, and therefore cannot have an edge, thus being determined as a non-edge (no false alarms); if the bitwise AND result is zero, then the existence of the edge needs to be verified through the adjacency list to ensure complete accuracy (no false alarms). The entire query process supports multi-core parallel processing because there are no dependencies between query operations.

[0019] Preferably, constructing a cache-optimized representation of graph data includes: Implemented using the GTWICE algorithm, the GT-vector of each vertex is initialized to all zeros, and a counter is initialized to record the number of times each vertex is selected into the maximum independent set, and k iterations are performed: S1: Sort all vertices in ascending order based on the counter value to generate a vertex sequence; S2: Based on the vertex sequence, use a greedy algorithm to calculate the i-th largest independent set; S3: Set the i-th position of the GT-vector corresponding to the vertex belonging to the largest independent set to 1, and increment the counter.

[0020] Specifically, the GTWICE algorithm uses a dynamic priority mechanism (ascending order sorting by counter) to ensure that the extracted k independent sets can cover the non-edges in the graph to the greatest extent. Its total time complexity is O(k(|V|+|E|)), which can efficiently handle ultra-large-scale graphs with billions of edges.

[0021] Preferably, the time complexity of the GTWICE algorithm is O(k(|V|+|E|)), where |V| is the number of vertices, |E| is the number of edges, and k is the predefined bit vector length.

[0022] Preferably, the predefined bit vector length is a multiple of 64 bits.

[0023] Preferably, the bitwise AND operation is completed within a single CPU cycle.

[0024] Preferably, the adjacency list in memory is queried using a binary search algorithm.

[0025] Preferably, the method supports multi-core parallel processing of multiple edge query requests without dependencies.

[0026] Specifically, such as Figure 2 As shown, taking a social network graph G containing 16 vertices as an example, GT-vectors are constructed for it using the GTWICE algorithm (k=8), resulting in the following: Figure 3 The results shown are partial.

[0027] Let's take the query (A, D) as an example to illustrate the processing procedure: (1) Cache read: The CPU reads Ĝ(A)=10011000 and Ĝ(D)=10000001 from the L3 cache; (2) Bitwise operations: Calculate 10011000 & 10000001 = 10000000 (the result is non-zero); (3) Decision: According to the independent set theory, immediately determine that (A,D) is not an edge, and the query ends; (4) Performance comparison: The traditional adjacency list scheme needs to read the neighbor list of vertex A from memory (which may trigger a cache miss) and perform a binary search in the list to confirm that D does not exist. The whole process takes tens of nanoseconds. This invention reduces it to 1 CPU cycle and has no memory access.

[0028] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.

Claims

1. A graph data processing method based on cache optimization, characterized in that, include: To construct a cache-optimized representation of graph data, each vertex in the graph is assigned a bit vector GT-vector of fixed length k, where each bit indicates whether the vertex belongs to a pre-computed maximum independent set; The adjacency list representation of the graph data is preserved and stored in memory as the base data; For edge query requests, read the GT-vectors of the two vertices from the CPU cache and perform a bitwise AND operation; If the bit AND result is non-zero, it is determined to be a non-edge and the first Boolean result indicating that the edge does not exist is immediately returned; If the bitwise AND result is zero, the adjacency list in memory is queried for verification, and a second Boolean result indicating whether the edge exists or not is returned.

2. The graph data processing method based on cache optimization according to claim 1, characterized in that, The cache-optimized representation of the constructed graph data includes: Implemented using the GTWICE algorithm, the GT-vector of each vertex is initialized to all zeros, and a counter is initialized to record the number of times each vertex is selected into the maximum independent set, and k iterations are performed: S1: Sort all vertices in ascending order according to the value of the counter to generate a vertex sequence; S2: Based on the vertex sequence, use a greedy algorithm to calculate the i-th largest independent set; S3: Set the i-th position of the GT-vector corresponding to the vertex belonging to the largest independent set to 1, and increment the counter.

3. The graph data processing method based on cache optimization according to claim 2, characterized in that, The time complexity of the GTWICE algorithm is O(k(|V|+|E|)), where |V| is the number of vertices, |E| is the number of edges, and k is the predefined bit vector length.

4. The graph data processing method based on cache optimization according to claim 3, characterized in that, The predefined bit vector has a length that is a multiple of 64 bits.

5. The graph data processing method based on cache optimization according to claim 1, characterized in that, The bitwise AND operation is completed within a single CPU cycle.

6. The graph data processing method based on cache optimization according to claim 1, characterized in that, The adjacency list in memory is queried using a binary search algorithm.

7. The graph data processing method based on cache optimization according to claim 1, characterized in that, The method supports multi-core parallel processing of multiple edge query requests without dependencies.

Images