BFS algorithm-oriented graph processing system caching method, controller and system

Abstract

The application discloses a graph processing system cache method, a controller and a system for a BFS algorithm, belongs to the technical field of performance optimization of a graph processing system, and comprises the following steps: based on the relationship between the access heat of a cache page and the number of unvisited vertices in the BFS algorithm, more accurate heat grading management is implemented on the cache page, when the cache is full, the page with low access potential is effectively eliminated, and the cache hit rate is improved; fragmented effective edge data is extracted from the cache page and separately stored in a cache managed in a fine granularity; by acquiring the proportion of the edge list of the vertex in the BFS algorithm task queue that has been resident in the cache, and in combination with the vertex out-degree and other attributes, the scheduling order of the vertex in the task queue is adjusted, and the data that has been resident in the cache is maximally utilized. The application can effectively improve the cache hit rate of the graph processing system when processing the BFS algorithm task, and further realizes the improvement of the access performance of the graph processing system.

Description

Technical Field

[0001] This invention belongs to the field of graph processing system performance optimization technology, and more specifically, relates to a caching method, controller and system for graph processing systems oriented towards BFS-like algorithms. Background Technology

[0002] Breadth-First Search Like Algorithms (BFS-like Algorithms), such as BFS, WCC, and APSP, are a class of graph algorithms based on the Breadth-First Search (BFS) concept. They start with one or more initial vertices and recursively traverse their neighboring vertices, processing them in a hierarchical order. Because their level-by-level traversal directly addresses the needs of finding the shortest path and detecting connectivity in graphs, they are widely used in social network analysis, real-time recommendation systems, bioinformatics modeling, and intelligent transportation optimization. BFS-like algorithms need to determine and activate the vertices to be processed in the next stage during iteration; these vertices are called active vertices. During the processing of active vertices, the system loads the edge list of active vertices from external memory so that the BFS-like algorithm can update vertex attributes based on the loaded edge data and generate a new set of active vertices. This process continues until all graph data has been traversed.

[0003] To optimize the management and read / write operations of vertices and edges in graph structures, enabling the efficient execution of various graph algorithms, including BFS-like algorithms, graph processing systems have been researched and developed. With the advent of the big data era, the scale of graph data has expanded dramatically, making it difficult for graph processing systems that originally relied solely on memory for data storage to cope effectively. Therefore, people began to utilize external storage to store massive amounts of graph data at low cost for graph processing. Given the extremely complex relationships between vertices (i.e., edges) in real-world graphs, and the fact that the number of edges is usually far greater than the number of vertices, graph processing systems often choose to store large-sized edge data in external storage to achieve low-cost storage of large-scale graphs, while storing smaller-sized vertex data in faster memory to achieve a balance between storage cost and access speed. This type of graph processing system is called a semi-external storage graph processing system.

[0004] To provide better graph processing performance, semi-external memory graph processing systems allocate a portion of user space memory as a graph system cache to store frequently accessed or recently used edge data. The graph system cache is managed at the page level (typically 4KB), with one page storing the edge list for multiple vertices. Given the significant difference in access latency between external storage and main memory (e.g., DRAM memory access speeds reach nanosecond levels, while external storage access speeds, represented by SSDs, are typically microsecond levels), efficiently utilizing the limited cache space of the graph system to improve hit rates becomes crucial for optimizing graph processing performance.

[0005] However, caching in graph processing systems oriented towards BFS-like algorithms faces the problem of low cache hit rates. This is mainly due to two aspects: First, regarding data popularity, traditional replacement strategies cannot reasonably retain hot cache pages for BFS-like algorithms. Traditional replacement strategies are based on the principle of temporal locality, meaning that hot data, once accessed, is likely to be accessed again in the future. However, in graph processing systems oriented towards BFS-like algorithms, cached data does not possess this temporal locality characteristic, making it difficult to retain hot cache pages in the graph system cache, resulting in a low hit rate. Second, regarding the proportion of effective data, graph edge data has a finer granularity than cached pages, and the physical storage location of active vertices often exhibits spatial discreteness. When loading edge data of active vertices into the graph cache at the page granularity, the system simultaneously loads inactive and invalid data, resulting in a low proportion of effective data in the cached pages. Because cache space is occupied by redundant data, in situations where cache space resources are scarce, truly needed effective data is replaced prematurely, leading to a significant decrease in cache hit rate. Summary of the Invention

[0006] To address the shortcomings and improvement needs of existing technologies, this invention provides a caching method, controller, and system for graph processing systems oriented towards BFS-like algorithms. The aim is to improve the cache hit rate of graph processing systems when processing BFS-like algorithm tasks, thereby enhancing the access performance of graph processing systems.

[0007] To achieve the above objectives, according to one aspect of the present invention, a caching method for a graph processing system oriented towards a BFS-like algorithm is provided, comprising: a cache initialization step and a cache insertion step. The cache initialization steps include: allocating a main cache for caching pages within the system cache, and maintaining a first mapping table and a resident value bucket linked list; a page is used to store an edge list with multiple vertices; the first mapping table records the mapping relationship between the page's external memory address and the cache address; in the resident value bucket linked list, the first... n Bucket # is used to store resident value located at [ nk , ( n+ 1) k-Page metadata for the range 1] k The value is a preset positive integer; the dwell value of a page is the number of remaining vertices on the page; the metadata includes the number of remaining vertices and cached page pointers; The cache insertion steps include: S1: Retrieve a free page P from the main cache. c ; S2: Cache the edge list in page P to be cached. c In the middle, and link page P with page P c The mapping relationship between them is recorded in the first mapping table; S3: If the value bucket in the linked list is the first If bucket number 1 already exists, proceed to S4; otherwise, create bucket number 2 in the resident value bucket linked list. Bucket #1, transfer to S4; F Page P c The number of remaining vertices, Indicates rounding down; S4: Generate page P c The metadata is then inserted into the resident value bucket linked list. In the bucket number.

[0008] Furthermore, in the resident value bucket linked list, the nodes in each bucket are stored in ascending order of resident value; Furthermore, S1 includes: Retrieve a free page from the main cache. If successful, record it as page P. c Otherwise, trigger the page eviction step to free up a free page, which is then denoted as page P. c ; The page eviction process includes: Evict the cached page corresponding to the first node in the first bucket of the resident value bucket linked list from the main cache to release the corresponding cached page, and delete the node from the resident value bucket linked list.

[0009] Furthermore, the initialization steps also include: allocating a residual edge cache in the system cache for caching edges, and maintaining a second mapping table; the second mapping table is used to record the mapping relationship between the external memory address and the cache address of the edge. Furthermore, the page eviction process also includes: When evicting a cached page from the main cache, if the proportion of unaccessed vertices is lower than a preset threshold M, the ratio between the number of accessed edges in the page to be evicted and the page capacity is calculated. If the ratio exceeds a preset threshold T, the unaccessed edges in the page to be evicted are inserted into the residual edge cache.

[0010] Furthermore, the edges are stored in the residual edge cache in the form of a circular queue; Furthermore, the residual edge cache for unvisited edges in the page to be evicted includes: If the remaining space in the circular queue is sufficient to accommodate the edge to be inserted, then the edge to be inserted is inserted into it, and the mapping from the external memory address to the cache address of these edges is recorded in the second mapping table; If the remaining space in the circular queue is insufficient to accommodate the edge to be inserted, the position of the first edge that has been queried in the circular queue is located, and the edge is expelled from that position and the corresponding space is released until the remaining space in the circular queue is sufficient to accommodate the edge to be inserted; the edge to be inserted is inserted into the circular queue, and the mapping from the external memory address to the cache address of these edges is recorded in the second mapping table.

[0011] Furthermore, in the resident value bucket linked list, the nodes in each bucket are organized through a doubly linked list.

[0012] Furthermore, the graph processing system caching method for BFS-like algorithms provided by the present invention further includes: a cache page query step; The steps for querying a cached page include: The first mapping table is queried to obtain the mapping relationship between the external storage address and the cache address of the page to be queried. If the query is successful, it is determined that the main cache has been hit and the page is read from the main cache; otherwise, it is determined that the main cache has not been hit, the page is read from the external storage and inserted into the cache through the cache insertion step.

[0013] Furthermore, the caching method for graph processing systems oriented towards BFS-like algorithms provided by this invention also includes: After each cached page is read, the remaining vertex count of that page is decremented by 1 to update the page's metadata; If the number of remaining vertices on the page is updated... And the first If bucket number 1 exists, the updated metadata will be transferred from bucket number 2. Delete from bucket # and insert from bucket #. Numbered barrels; If the number of remaining vertices on the page is updated... And the first If bucket number 1 does not exist, create bucket number 2 in the resident value bucket linked list. After bucket number 1, the updated metadata will be transferred from bucket number 2. Delete from bucket # and insert from bucket #. Bucket number; in, This represents the number of remaining vertices after the update. This indicates the number of remaining vertices before the update.

[0014] Furthermore, the caching method for graph processing systems oriented towards BFS-like algorithms provided by this invention also includes: For each active vertex in the list of active vertices, calculate the proportion of its visited vertices to all its neighboring vertices, and use this as the grouping index for the corresponding active vertex. Sort the list of active vertices in ascending order of their grouping indices to obtain an ordered list of active vertices. Calculate the grouping index difference between adjacent vertices in the ordered list of active vertices to obtain the difference array; The median of the grouped index differences is multiplied by a preset multiple to obtain the grouping threshold. Select the differences in the difference array that exceed the grouping threshold and are local maxima as the split points; The list of ordered active vertices is divided into multiple groups based on each dividing point; Calculate the scheduling score for each group and set the scheduling priority for each group based on the scheduling score; the scheduling score is the reciprocal of the sum of the proportion of the edge list of vertices in the group that has been cached and the out-degree; the lower the scheduling score, the higher the scheduling priority of the group.

[0015] According to another aspect of the present invention, a cache controller for a graph processing system oriented towards a BFS-like algorithm is provided, comprising: A computer-readable storage medium for storing computer programs; And a processor for reading a computer program stored in a computer-readable storage medium to implement the above-described graph processing system caching method for the BFS algorithm provided by the present invention.

[0016] According to another aspect of the present invention, a graph processing system oriented towards a BFS-like algorithm is provided, comprising: external storage, a cache, and the above-described cache controller for a BFS-like graph processing system provided by the present invention.

[0017] In summary, the above-described technical solutions conceived in this invention can achieve the following beneficial effects: (1) This invention predicts the popularity level of a page by counting the number of source vertices connected to unvisited edges in the cached page, and constructs a bucket table structure accordingly. Cache pages with similar popularity levels are assigned to the same bucket for management, making the cache replacement strategy more suitable for the graph cache popularity characteristics oriented towards BFS-like algorithms. As a result, hot cache pages can be retained in the graph processing system cache, effectively improving the cache hit rate.

[0018] (2) In the resident value bucket linked list maintained by the present invention, the resident value range corresponding to each bucket is ordered, and the nodes in each bucket are ordered according to the resident value of the page. Thus, the cached page corresponding to the first node in the first bucket of the resident value bucket linked list is the page with the lowest resident value. Therefore, the present invention can quickly locate the page with the lowest resident value when the cache is full, and the page with greater future access potential can reside in the cache to improve the cache hit rate.

[0019] (3) In addition to the page cache (i.e. the main cache), the present invention also maintains a fine-grained residual edge cache and dynamically monitors the dispersion of the data to be accessed in the obsolete page. The severely fragmented data to be accessed is taken out of the page and placed separately in the residual edge cache, which can increase the proportion of effective cache data and further improve the cache hit rate.

[0020] (4) In this invention, a circular queue is used to store edges in a fine-grained residual edge cache. When the circular queue is full, the position of the first accessed edge is located, and from that position, the edge is expelled and the corresponding space is released until enough space is released. This management strategy effectively reduces the additional overhead caused by the increase of objects in fine-grained management, realizes lightweight management, and achieves a balance between improving cache hit rate and increasing management overhead.

[0021] (5) This invention monitors the proportion of edge lists of active vertex neighbors stored in the cache, and analyzes the proportion of effective data in the cached pages in real time by combining vertex attributes, thereby optimizing the scheduling priority of the task queue in the algorithm. This enables the maximum reading of content already residing in the cache, especially pages with a large proportion of effective data, reducing the probability that data in the cache will be replaced before it is read, thereby further improving the cache hit rate. Attached Figure Description

[0022] Figure 1 This is a schematic diagram of a caching method for a graph processing system oriented towards BFS provided in an embodiment of the present invention.

[0023] Figure 2 This is a schematic diagram of a vertex access state-driven replacement strategy provided in an embodiment of the present invention.

[0024] Figure 3 A flowchart illustrating the cache page query logic of the vertex access state-driven replacement strategy provided in this embodiment of the invention.

[0025] Figure 4 This is a flowchart of the cache page insertion logic of the vertex access state-driven replacement strategy provided in this embodiment of the invention.

[0026] Figure 5 This is a flowchart of the cache page reading logic of the vertex access state-driven replacement strategy provided in an embodiment of the present invention.

[0027] Figure 6 This is a schematic diagram of the data management structure for the residual edge fine-grained cache management strategy provided in an embodiment of the present invention.

[0028] Figure 7 This is a flowchart of the admission mechanism for the fine-grained cache management strategy for residual edges provided in this embodiment of the invention.

[0029] Figure 8 This is a flowchart of the adaptive median gap segmentation algorithm based on a cache-aware task scheduling strategy provided in this embodiment of the invention. Detailed Implementation

[0030] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0031] In this invention, the terms "first," "second," etc. (if present) in the invention and the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0032] Currently, massive amounts of information in fields such as social network analysis, real-time recommendation systems, and intelligent traffic optimization are abstracted into graph structures, and their analysis often relies on graph traversal operations. Therefore, Breadth-First Search Like Algorithms (BFS-like algorithms) based on graph traversal characteristics are widely used in graph processing systems. However, graph processing systems based on BFS-like algorithms suffer from low cache hit rates, leading to poor system access performance. The reasons for this deficiency are mainly twofold. First, the popularity of cached pages accessed by BFS-like algorithms is related to the number of unvisited vertices on the page. Traditional cache replacement strategies, which rely on popularity characteristics, are not compatible with this, resulting in a low cache hit rate. Second, the granularity of graph edge data is usually smaller than that of cached pages, and the edge data accessed in each iteration of BFS-like algorithms is discretely distributed across different pages. This introduces invalid data with each cache page loading operation, which crowds out cache space, ultimately leading to a significant decrease in cache hit rate. Therefore, researching efficient management methods for graph system caching to improve cache hit rate is of great significance.

[0033] To address the aforementioned issues, this invention provides a caching method, controller, and system for graph processing systems based on a BFS-like algorithm. It optimizes the cache replacement strategy, specifically by implementing a vertex access state-driven replacement strategy. Based on the relationship between the access popularity of cached pages and the number of unvisited vertices in a BFS-like algorithm, it implements more precise hierarchical management of cached pages based on their access frequency. When the cache is full, pages with low access potential are effectively evicted to improve the cache hit rate. Furthermore, this invention optimizes the granularity management strategy and task scheduling strategy.

[0034] The following is an example.

[0035] Example 1: A caching method for graph processing systems based on BFS-like algorithms, such as... Figure 1 As shown.

[0036] In this embodiment, during system initialization, a main cache for caching pages is allocated from the system cache. Regarding the replacement strategy for the main cache, the aforementioned vertex access state-driven replacement strategy is implemented. This strategy uses the number of remaining unvisited vertices on the page as the cache retention value to measure the future access frequency of the page. For example... Figure 2 As shown, this replacement strategy maintains a linked list of cached value buckets. Each bucket corresponds to a cached value range, and cached pages with values ​​belonging to that range are placed in the same bucket. To facilitate page lookup, the cache uses a hash table to store the mapping between edge data addresses and cached page addresses. Cached pages are managed using a method that separates metadata and page data. The metadata node contains pointers to the page data. Each cached page's metadata node needs to maintain a remaining vertex counter, representing the number of source vertices belonging to the remaining unvisited edge list of the current page. Cached pages in the same bucket use a doubly linked list to organize their metadata nodes to facilitate node insertion and deletion. When a cached page hits and accesses the edge list of remaining vertices, the remaining vertex counter is updated. It is easy to understand that the count is set to a preset positive integer. k Let each bucket represent the range of residency value corresponding to it. Then, in the residency value bucket linked list, the first... n Bucket # is used to store resident value located at [ nk ,( n+ 1) k- Page metadata for the 1] range.

[0037] Optionally, such as Figure 2 As shown, in this embodiment, the dwell value range is set to 3. Therefore, pages with remaining vertex counts of 0, 1, and 2 belong to bucket 0, pages with remaining vertex counts of 3, 4, and 5 belong to bucket 1, and so on.

[0038] After implementing the data structure for the vertex state-driven replacement strategy, the next step is to implement the processing logic of the cache replacement strategy during specific read and write processes, including cache page query logic, cache insertion logic, cache page eviction logic, and cache read logic.

[0039] The cached page query logic is as follows Figure 3 As shown. The main steps are as follows: (1) The system will query the cache based on the address of the vertex edge list. Like most traditional caches, the mapping from vertex edge list address to cache page location is maintained by a hash table. If the mapping corresponding to the address can be found in the hash table, the cache is hit; otherwise, it is not hit. (2) Execute cache insertion logic and cache read logic.

[0040] The flowchart of the cache insertion logic is as follows: Figure 4 As shown, the main steps are as follows: (1) Obtain a free page from the remaining cache space; (2) Take the high 52 bits of the vertex edge list address and construct a hash relationship with the new cache page address, then store it in the cache's metadata table. The low 12 bits of the vertex edge list address are used as its offset in the cache page; (3) Get the number of unread vertices F on the page and initialize the remaining vertex counter of the cached page; (4) If node F / k exists, insert the cache after node F / k in the cache-resident value list. If node F / k does not exist, create a new F / k list in the cache-resident value list. (5) Check the cache capacity. If the cache is full, execute the page eviction logic and evict the first cache page in the linked list with the lowest cache value.

[0041] When a cache access is hit, or when the system retrieves a page from external storage and puts it into the cache, the cache read logic is triggered to read valid data from the cached page for use by a BFS-like algorithm. The cache read logic is as follows: Figure 5 As shown. The main steps are as follows: (1) Each time a cached page is read, the edge list of a vertex is accessed, so the remaining vertex counter of the page is decremented by 1; before and after the update, the value of the remaining vertex counter of the page is respectively... and express; (2) Next, check the number of remaining vertices. Does it meet the requirements? If the conditions are met, then the page should be redirected to... The linked list is below the number. If... If the sign is not 0 and the linked list does not exist, then the linked list must be created. If the value is less than or equal to 0, the page will be evicted.

[0042] The replacement strategy based on vertex access state proposed in this embodiment predicts the popularity level of a page by counting the number of source vertices connected to unvisited edges in the cached page. Based on this, a bucket table structure is constructed, and cached pages with similar popularity levels are assigned to the same bucket for management. This makes the cache replacement strategy more suitable for the popularity characteristics of graph caches oriented towards BFS-like algorithms. When the cache capacity is about to be full, the cache replacement strategy will prioritize the replacement of cached pages in low-population buckets, so that pages with greater future access potential can remain in the cache, thereby improving the cache hit rate.

[0043] As a further preferred implementation, this embodiment further optimizes the granularity of cache management by extracting fragmented valid edge data from cached pages and storing it separately in a fine-grained cache. This increases the proportion of valid data on the page, thereby improving the cache hit rate. This fine-grained cache is a residual edge cache allocated from the system cache for caching edges.

[0044] like Figure 6 As shown, the fine-grained cache uses a FIFO strategy to maintain a circular queue to manage fine-grained data in the cache. Each object in the circular queue is associated with an access bit. When a residual edge cache hits an object, this access bit is used to mark it as accessed. To make good use of the object's access information during eviction and to reduce the complexity of the operation, the residual edge cache eviction mechanism is designed as follows: When the cache capacity is full, the queue pointer finds the first node marked as accessed. From this point onwards, regardless of whether subsequent nodes have been accessed, they will be evicted until enough space is made for the newly added element. Figure 6 In the cache management strategy shown, when two new nodes are inserted and the cache capacity reaches the upper limit, the queue pointer first starts to evict from the first access mark of the linked list that is 1. It can evict nodes regardless of whether the subsequent access mark is 1, until space for two nodes is freed up for the new nodes to store.

[0045] like Figure 7 As shown, a fine-grained management strategy requires an admission mechanism to determine which fragmented, valid data should be migrated to the fine-grained cache. The steps are as follows: (1) The system periodically scans the visited vertex array of the BFS-like algorithm to monitor the proportion of untraversed vertices in real time. When this proportion is lower than the preset threshold M, it enters the convergence period triggering stage and proceeds to step (2) to start the detection of pages to be evicted in the main cache; (2) When the main cache is about to evict a page, the reading status of the edge data contained in the evicted page is counted, and the usage status of the edge data is tracked through the bitmap marking mechanism—each binary bit in the bitmap corresponds to an edge (0 indicates unvisited, 1 indicates visited). It is easy to understand that if the source vertex of an edge has been read (i.e., it is in an active state), then its edge list must have been traversed. At this time, the edge is marked as 1 in the bitmap; (3) If the total size of the edges traversed on the page to be eliminated exceeds a certain proportion T of the page capacity, then the untraversed edges are inserted into the residual edge cache. Otherwise, continue monitoring the page to be eliminated in step (2).

[0046] In terms of cache granularity management strategy, this embodiment dynamically monitors the dispersion of data to be accessed in evicted pages, and retrieves highly fragmented data to be accessed from the page and places it separately in the fine-grained cache, which can increase the proportion of effective cached data. To reduce the additional overhead caused by the increase in objects in fine-grained management, a lightweight management system based on a first-in-first-out queue is used to achieve a balance between improving cache hit rate and increasing management overhead.

[0047] As a further preferred implementation, this embodiment proposes a cache scoring mechanism to adjust the scheduling order of vertices in the task queue, so as to maximize the utilization of data already residing in the cache. Figure 8 As shown, the cache scoring mechanism uses an adaptive median gap partitioning algorithm to group active vertices in the task queue. This grouping strategy groups vertices with similar neighbor IDs together, meaning vertices with similar physical storage are grouped together. The specific steps of the adaptive median gap partitioning algorithm are as follows: (1) Sorting array: First, the proportion of visited vertices to all their neighboring vertices is used as the grouping index of the corresponding active vertices. The current active vertex array is sorted in ascending order according to the grouping index; the time complexity is O(nlogn). (2) Calculate the difference between adjacent elements: Traverse the sorted array and calculate the difference (interval) between adjacent elements to obtain the difference array; the time complexity is O(n); (3) Determine the segmentation threshold: Use the median of the differences multiplied by a factor (e.g., 3 times) as the grouping threshold; (4) Identify the split point: Traverse the difference array and find the difference position that exceeds the threshold and is a local maximum value, and use it as the split point for grouping; (5) Generate groups: Divide the sorted array into multiple subarrays according to the split point, and each subarray is a group.

[0048] This embodiment groups currently active vertices using the steps described above. Then, for each vertex within a group, it calculates the edge list cache coverage rate, which is the proportion of the edge list already cached. Next, it calculates the out-degree of each vertex. The edge list cache coverage rate and out-degree are added together, and the reciprocal is taken to obtain the scheduling score for that group. Groups with lower scores have higher scheduling priority. This method maximizes the reading of data already residing in the cache, reducing the number of times cached data is repeatedly replaced, thereby improving the cache hit rate.

[0049] Overall, this embodiment can provide a replacement strategy that adapts to the popularity characteristics of cached pages, thereby improving the cache hit rate of the graph system; it can provide a lightweight, fine-grained cache management method to increase the proportion of effective data on cached pages; and it can provide a cache information-aware task scheduling strategy to maximize the reading of content already residing in the cache, ultimately effectively improving the access performance of the graph processing system and increasing the cache hit rate.

[0050] Example 2: A cache controller for a graph processing system oriented towards a BFS-like algorithm, comprising: A computer-readable storage medium for storing computer programs; And a processor for reading a computer program stored in a computer-readable storage medium to implement the graph processing system caching method for the BFS algorithm provided in Embodiment 1 above.

[0051] Example 3: A graph processing system oriented towards a BFS-like algorithm includes: external storage, a cache, and a BFS-like graph processing system cache controller provided in Embodiment 2 above.

[0052] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A caching method for a graph processing system oriented towards a BFS-like algorithm, characterized in that, include: Cache initialization steps and cache insertion steps: The cache initialization steps include: allocating a main cache for caching pages in the system cache, and maintaining a first mapping table and a resident value bucket linked list; the page is used to store an edge list with multiple vertices; the first mapping table is used to record the mapping relationship between the page's external memory address and the cache address; in the resident value bucket linked list, the first... n Bucket # is used to store resident value located at [ nk , ( n+ 1) k- Page metadata for the range 1] k The value is a preset positive integer; the dwell value of a page is the number of remaining vertices on the page; the metadata includes the number of remaining vertices and cached page pointers; The cache insertion step includes: S1: Obtain a free page P from the main cache. c ; S2: Cache the edge list in page P to be cached. c In the middle, and link page P with page P c The mapping relationship between them is recorded in the first mapping table; S3: If the resident value bucket chain list of the first If bucket number 1 already exists, proceed to step S4; otherwise, create bucket number 2 in the resident value bucket linked list. Bucket #1, transfer to S4; F Page P c The number of remaining vertices, Indicates rounding down; S4: Generate page P c The metadata is then inserted into the resident value bucket linked list. In the bucket number.

2. The caching method for a graph processing system oriented towards a BFS-like algorithm as described in claim 1, characterized in that, In the resident value bucket linked list, the nodes in each bucket are stored in ascending order of resident value; Furthermore, S1 includes: Retrieve a free page from the main cache. If successful, record it as page P. c Otherwise, trigger the page eviction step to free up a free page, which is then denoted as page P. c ; The page removal step includes: The cached page corresponding to the first node in the first bucket of the resident value bucket linked list is evicted from the main cache to release the corresponding cached page, and the node is deleted from the resident value bucket linked list.

3. The caching method for a graph processing system oriented towards a BFS-like algorithm as described in claim 2, characterized in that, The initialization step further includes: allocating a residual edge cache in the system cache for caching edges, and maintaining a second mapping table; the second mapping table records the mapping relationship between the external storage address and the cache address of the edge. Furthermore, the page eviction step also includes: When a cached page is evicted from the main cache, if the proportion of unaccessed vertices is lower than a preset threshold M, the ratio between the number of accessed edges in the page to be evicted and the page capacity is calculated; if the ratio exceeds a preset threshold T, the unaccessed edges in the page to be evicted are inserted into the residual edge cache.

4. The caching method for a graph processing system oriented towards a BFS-like algorithm as described in claim 3, characterized in that, The edges are stored in the residual edge cache in the form of a circular queue; Furthermore, inserting unaccessed edges from the page to be evicted into the residual edge cache includes: If the remaining space in the circular queue is sufficient to accommodate the edge to be inserted, then the edge to be inserted is inserted into it, and the mapping from the external memory address to the cache address of these edges is recorded in the second mapping table; If the remaining space in the circular queue is insufficient to accommodate the edge to be inserted, the position of the first edge that has been queried in the circular queue is located, and the edge is expelled from that position and the corresponding space is released until the remaining space in the circular queue is sufficient to accommodate the edge to be inserted; the edge to be inserted is inserted into the circular queue, and the mapping from the external memory address to the cache address of these edges is recorded in the second mapping table.

5. The caching method for a graph processing system oriented towards a BFS-like algorithm as described in claim 2, characterized in that, In the resident value bucket linked list, the nodes in each bucket are organized through a doubly linked list.

6. The caching method for a graph processing system based on a BFS-like algorithm as described in any one of claims 1 to 5, characterized in that, Also includes: Steps for querying cached pages; The cached page query steps include: The first mapping table is queried to obtain the mapping relationship between the external storage address and the cache address of the page to be queried. If the query is successful, it is determined that the main cache has been hit and the page is read from the main cache; otherwise, it is determined that the main cache has not been hit, the page is read from the external storage and inserted into the cache through the cache insertion step.

7. The caching method for a graph processing system oriented towards a BFS-like algorithm as described in claim 6, characterized in that, Also includes: After each cached page is read, the remaining vertex count of that page is decremented by 1 to update the page's metadata; If the number of remaining vertices on the page is updated... And the first If bucket number 1 exists, the updated metadata will be transferred from bucket number 2. Delete from bucket # and insert from bucket #. Numbered barrels; If the number of remaining vertices on the page is updated... And the first If bucket number 1 does not exist, then create bucket number 2 in the resident value bucket linked list. After bucket number 1, the updated metadata will be transferred from bucket number 2. Delete from bucket # and insert from bucket #. Numbered barrels; in, This represents the number of remaining vertices after the update. This indicates the number of remaining vertices before the update.

8. The caching method for a graph processing system based on a BFS-like algorithm as described in any one of claims 1 to 5, characterized in that, Also includes: For each active vertex in the list of active vertices, calculate the proportion of its visited vertices to all its neighboring vertices, and use this as the grouping index for the corresponding active vertex. The list of activated vertices is sorted in ascending order of grouping index to obtain an ordered list of activated vertices. Calculate the grouping index difference between adjacent vertices in the ordered active vertex list to obtain the difference array; The median of the grouped index differences is multiplied by a preset multiple to obtain the grouping threshold. The differences in the difference array that exceed the grouping threshold and are local maxima are selected as the split points; The ordered list of active vertices is divided into multiple groups based on each segmentation point; Calculate the scheduling score for each group and set the scheduling priority for each group based on the scheduling score; the scheduling score is the reciprocal of the sum of the proportion of the edge list of vertices in the group that has been cached and the out-degree; the lower the scheduling score, the higher the scheduling priority of the group.

9. A cache controller for a graph processing system oriented towards a BFS-like algorithm, characterized in that, include: A computer-readable storage medium for storing computer programs; And a processor for reading a computer program stored in the computer-readable storage medium to implement the graph processing system caching method for BFS-oriented algorithm as described in any one of claims 1 to 8.

10. A graph processing system oriented towards a BFS-like algorithm, characterized in that, include: External storage, cache, and the cache controller for a BFS-oriented graph processing system as described in claim 9.

Images