Graph computing method and device, electronic equipment and storage medium

Abstract

The application provides a graph computing method and device, electronic equipment and storage medium, and relates to the technical field of data processing. The graph computing method comprises the following steps: determining a node task corresponding to a graph computing task and a thread pool corresponding to the node task according to real-time system resources and task information of the graph computing task, so that the thread pool acquires exchange data of other node tasks and node cache data in a cache module; in the case where the node cache data is not acquired, a preset database is searched through an asynchronous mode, and the searched node cache data is written into the cache module, so that the thread pool is re-acquired based on a synchronous mode; in the case where the node cache data is acquired, multi-thread synchronous calculation is performed according to the node cache data and the exchange data, so as to determine a graph computing result. The application combines synchronous processing, asynchronous processing and the mode of using a preset database when a cache is not hit, thereby improving the overall efficiency in large-scale data processing.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, for example to a graph computing method and device, electronic equipment and storage medium. BACKGROUND

[0002] Graph computing is a computing mode based on graphs (data structures composed of vertices and edges) to process and analyze data in the graphs. After rapid development, domestic environment has realized application landing in many fields, but compared with foreign hardware platforms, there are still problems such as insufficient performance of single-core processors, insufficient efficiency of multi-core cooperation, etc. On the multi-core architecture, although the number of cores of domestic processors is increasing, there is still a gap in efficiency, data transmission speed and task scheduling optimization when the multi-core works cooperatively, compared with foreign processors. For some complex computing tasks, some multi-core processors can more efficiently allocate tasks, share data, fully exert the performance of each core, and realize a substantial improvement in overall performance, while domestic processors still have room for improvement in this regard.

[0003] At present, the traditional graph system can only rely on the task submission mode of Spark (Scalable Processing Engine for Large-Scale Data Analytics, a fast general-purpose computing engine for large-scale data analytics) to read the point and edge data stored in the external storage, while Spark itself is a complex distributed computing framework, and the task submission process involves multiple links, including task initialization, parameter configuration, resource application and allocation, etc. These links will generate additional overheads, such as the need to load various configuration files and dependent libraries during task initialization, errors in parameter configuration may cause the task to be unable to execute correctly, and resource application and allocation may be delayed due to tight cluster resources or unreasonable allocation strategy, thereby causing low efficiency of graph computing in large-scale data processing. SUMMARY

[0004] The present application aims to provide a graph computing method and device, electronic equipment and storage medium.

[0005] According to an aspect of the present application, a graph computing method is provided, comprising: determining, according to real-time system resources and task information of a graph computing task, a node task corresponding to the graph computing task and a thread pool corresponding to the node task, so that the thread pool acquires exchange data of other node tasks and node cache data in a cache module; in a case where the node cache data is not acquired, searching a preset database in an asynchronous manner, and writing the searched node cache data into the cache module, so that the thread pool reacquires based on a synchronous manner; in a case where the node cache data is acquired, performing multi-thread synchronous calculation according to the node cache data and the exchange data, to determine a graph computing result.

[0006] According to an aspect of the present application, a graph computing device is provided, comprising:

[0007] a resource allocation module configured to determine, according to real-time system resources and task information of a graph computing task, a node task corresponding to the graph computing task and a thread pool corresponding to the node task, so that the thread pool acquires exchange data of other node tasks and node cache data in a cache module;

[0008] a data acquisition module configured to, in a case where the node cache data is not acquired, search a preset database in an asynchronous manner, and write the searched node cache data into the cache module, so that the thread pool reacquires based on a synchronous manner;

[0009] a calculation module configured to, in a case where the node cache data is acquired, perform multi-thread synchronous calculation according to the node cache data and the exchange data, to determine a graph computing result.

[0010] According to an aspect of the present application, an electronic device is provided, comprising: a processor; a memory storing a computer program, when the computer program is executed by the processor, causing the processor to execute the method as described above.

[0011] According to an aspect of the present application, a non-transitory computer readable medium is provided, storing readable instructions, when the instructions are executed by a processor, causing the processor to execute the method as described above.

[0012] It should be understood that the above general description and the following detailed description are only exemplary and do not limit the present application.

[0013] Advantages:

[0014] Through the above-mentioned embodiments provided by the present application, the graph computing task is fragmented by real-time sensing system resources, and thread pool and other resources are allocated to the obtained node tasks, so as to avoid delay due to tight cluster resources or unreasonable allocation strategy. Through the synchronous processing and asynchronous processing mode, the process of graph data lookup can be implemented, and the targeted calculation of different node tasks can be realized, while the data consistency can be ensured. The thread pool can obtain data from other nodes (exchange data) and cache modules (node cache data) at the same time, so as to reduce the network transmission overhead. When the cache is not hit, data is automatically loaded from the preset database and written into the cache module, so as to improve the subsequent task execution efficiency, and further improve the overall efficiency of graph computing in large-scale data processing. BRIEF DESCRIPTION OF DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art according to these drawings without departing from the scope of the present application.

[0016] Figure 1 The overall structure schematic diagram of the graph computing system provided by the embodiments of the present application is shown in the figure.

[0017] Figure 2 The flowchart of the graph computing method provided by the embodiments of the present application is shown in the figure.

[0018] Figure 3 The block diagram of the graph computing device provided by the embodiments of the present application is shown in the figure.

[0019] Figure 4 The structure schematic diagram of the electronic device provided by the embodiments of the present application is shown in the figure. DETAILED DESCRIPTION

[0020] Example embodiments now will be described more fully hereinafter with reference to the accompanying drawings. Example embodiments may, however, be implemented in many different forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the concept of example embodiments to those skilled in the art. Like reference numerals refer to like elements throughout the figures, and thus description of the same will be simplified or omitted.

[0021] Moreover, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a thorough understanding of embodiments of the application. One skilled in the relevant art will recognize, however, that the application can be practiced without one or more of the specific details, or with other methods, components, materials, and so forth. In other instances, well-known structures, devices, implementations, and operations have not been shown or described in detail to avoid obscuring aspects of the application.

[0022] The block diagrams in the drawings show only the functionality of the embodiments and do not imply that the embodiments will take the form discussed. For example, the functionality can be split into different blocks, some of which can be combined or eliminated, and other blocks can be created and / or divided. The embodiments described herein can be implemented in software and / or hardware. The software can comprise computer readable instructions stored on computer readable media such as computer memory, magnetic or optical storage media, or any other storage devices.

[0023] The flow diagrams depicted herein are merely examples and that numerous variations and modifications can be made to these diagrams without departing from the spirit and scope of the application. For instance, the operations can be performed in a variety of orders and over a variety of time frames. Also, an operation described, for example, as sequential in nature can be performed in parallel, and / or as a clustered operation, and vice versa depending on the functionality involved. For the most part, embodiments can be implemented as a routine (e.g., such as a routine for signal processing) that is stored in and executed by software, firmware, hardware, or a combination of software, firmware, and hardware from various companies.

[0024] It should be understood that although the terms first, second, third, etc. can be used herein to describe various components, these components should not be limited by these terms. These terms are used only to distinguish one component from another. Thus, a first component discussed below could be termed a second component without departing from the teachings of the present application. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0025] Figure 1The overall structure of the graph computing system provided by the embodiments of the present application is shown in the schematic diagram, and the present application can be applied to a domestic environment. The domestic environment generally refers to a domestic development environment in the fields of information technology, industrial manufacturing, scientific research, etc., dominated by technologies, products, standards and ecological systems independently researched and developed by China. The core goal is to achieve technology self-containment and reduce dependence on foreign countries, and to protect national security and industrial competitiveness. The system can be divided into three parts, namely a scheduling mechanism part, a cache mechanism part, and a part of a domestic operating system and a hardware platform. The domestic operating system and hardware platform can include the underlying graph database used by the currently developed operating system and hardware platform in China, i.e., the preset database in the present application. The scheduling mechanism includes multiple communication layers and multiple coroutine pools, wherein the communication layer can be implemented based on Brpc (Baidu Remote Procedure Call), and the coroutine pool can be implemented based on Folly (Facebook Open-source Library). The coroutine library (not shown in the figure) in the present application is a basic technical component that provides underlying interfaces such as coroutine creation, suspension and recovery; the coroutine pool is an encapsulated application of the coroutine library, which pre-creates coroutine instances and puts them into the pool for reuse (similar to a thread pool), reducing the overhead of coroutine creation. The coroutine tasks in the coroutine pool eventually rely on the threads of the thread pool for execution (coroutines cannot exist without threads). The thread pool allocates a physical thread, which schedules multiple coroutines (such as computing coroutines and communication coroutines) through the coroutine pool; the number of threads in the thread pool determines the maximum parallelism of the coroutine tasks (such as 8-core CPU thread pool executing up to 8 coroutine tasks in parallel).

[0026] As shown in Figure 1 The part of the cache mechanism has multiple cache modules, and the graph computing process includes multiple edges and nodes, each node can correspond to a cache module, and each cache module can be implemented based on CacheLib (Cache Library).

[0027] For each cache module, two operations of lookup and write can be performed. The hybrid cache lookup process for looking up graph data contains three processing methods. First, look up the graph data in the DRAM cache. If the graph data is found and not expired, call it; if it is expired, mark it and return that it is not found synchronously, like process 1. If the graph data does not exist in the DRAM cache, try to look it up in the NVM cache. If it is found, write it into the DRAM cache asynchronously, and then call it, like process 2. If the graph data is not found in the NVM cache, feedback the lookup result asynchronously, like process 3.

[0028] For the process of writing graph data, the graph data to be used is written into the DRAM cache. For data that meets the preset eviction condition, the data can be evicted from the DRAM cache to the NVM cache. This process can be implemented by using a write mechanism. The data stored in the NVM cache can also be subjected to cache eviction.

[0029] Based on the system Figure 1 When processing large-scale graph computing, the thread pool allocates N threads (such as N = CPU core number), each thread is associated with a coroutine pool, and M coroutines (M is greater than N) are maintained in the coroutine pool to realize the hybrid scheduling of “multi-thread parallel + single-thread multi-coroutine asynchronous” (the scene of “computing in one coroutine and communicating in another coroutine” is realized through this mechanism). The coroutine pool realizes asynchronous communication through the Brpc interface of the coroutine library (such as “coroutines exchange data with other nodes”), and the thread pool obtains data through the CacheLib cache mechanism (such as coroutines reading cache data from DRAM / NVM), which forms a technical closed loop with Brpc and CacheLib to optimize the performance of graph computing in the domestic environment.

[0030] The specific implementation manner can refer to the following embodiments.

[0031] Figure 2 A flowchart of a graph computing method provided by the embodiment of the present application is provided. The method of the embodiment can be applied to a graph computing server. As shown in Figure 2 , the method comprises steps S20, S21 and S22.

[0032] In step S20, according to real-time system resources and task information of a graph computing task, a node task corresponding to the graph computing task and a thread pool corresponding to the node task are determined, so that the thread pool obtains exchange data of other node tasks and node cache data in a cache module.

[0033] In this application, users may have multiple graph computation tasks as input, each involving multiple nodes, and the tasks corresponding to each node can be considered node tasks. Thread pools can be pre-configured for different node tasks. Real-time system resources can be used to characterize the current thread usage, memory usage, CPU utilization, etc., of the graph computation system. In the specific implementation process, the status of hardware and software resources can be monitored. Hardware resource monitoring can include CPU utilization: real-time collection of core loads (such as thread execution queue length and CPU time slice utilization) through the Foly thread pool; memory status: monitoring DRAM / NVM usage (such as remaining space and cache hit rate) using the CacheLib memory allocator; and storage and network: tracking the I / O throughput of SSD (Solid State Drive) / NVMe (Non-Volatile Memory Express), network latency and bandwidth usage of Brpc communication. Software resource status can include: Task queue depth: the number of tasks to be executed maintained by the scheduling mechanism, reflecting the current system load; Coroutine / thread status: the Foly coroutine library monitors the frequency of coroutine switching and the number of idle threads in the thread pool to determine the degree of idleness of computing resources.

[0034] In some implementations, real-time system resources can be directly detected and acquired. Task information for graph computation tasks can be sent by the user or pre-stored in the system and read and executed at the corresponding time. Node tasks and their corresponding thread pools can be extracted from the task information. The computation process of node A requires some data storage and caching modules and / or a pre-set database. Other data may be the computation results generated after the completion of tasks such as node B and node C, which may serve as intermediate results for node A. Therefore, the thread pool can obtain the node cache data related to that node task from the cache module based on the node task's situation. It can also communicate with thread pools of other nodes to obtain exchange data related to the solar term tasks.

[0035] In other implementations, regarding data caching in the caching module, each compute node needs to provide memory space (the space of the caching module) to store vertex data, such as vertex degrees (the in-degree and out-degree of vertices in a given slice when the task is sliced) and edge bitmaps (whether a vertex has an in-edge or out-edge). Furthermore, related edge information (the start point, end point, and weight of each edge) also needs to be stored. When the memory capacity is insufficient to hold the edge information, caching to high-speed storage devices such as SSDs should be considered.

[0036] In step S21, if no node cache data is obtained, the preset database is searched asynchronously, and the found node cache data is written to the cache module so that the thread pool can retrieve it again in a synchronous manner.

[0037] In this application, if the thread pool fails to retrieve cached node data during the process, it can asynchronously return the search result and search the preset database. After the search, it can then retrieve the data again using synchronous processing. Figure 1 As shown. Synchronous processing refers to tasks being executed sequentially, requiring the current operation to complete before proceeding to the next step. Asynchronous processing refers to tasks not blocking the current thread, but executing in the background via callbacks, events, or coroutines, and notifying the main program (executed by the graph computing server) upon completion. The preset database can be a pre-configured underlying graph database used to store graph data related to current domestic platforms, etc.

[0038] In some implementations, if the node cache data is not found in the cache module, an asynchronous method can be used to search the preset database and write the found node cache data into the cache module. When the node cache data is successfully written into the cache module, the thread pool can retrieve the node cache data from the cache module again using synchronous processing.

[0039] In step S22, if the node cache data is obtained, multi-threaded synchronous calculation is performed based on the node cache data and the exchange data to determine the graph calculation result.

[0040] In some implementations, if node cache data is successfully found in the caching module, the calculation process for that node task can be performed based on the node cache data and exchanged data, thus completing the entire graph computation task and obtaining the graph computation result. In this application, obtaining node cache data means that the node cache data is available and has not expired.

[0041] In this application, references Figure 1 The implementation method of the Foly thread pool:

[0042] The Folly library provides a thread pool implementation for convenient management and scheduling of threads. In this mechanism, the thread pool automatically allocates threads based on the number and priority of tasks, enabling parallel execution of distributed computing tasks. For example, a thread pool can be created using a pre-defined thread pool executor, with parameters such as the number of threads and queue size set. When a task is submitted, the thread pool selects a thread from its idle threads to execute it. If no idle threads are available, the task is placed in a task queue to await execution. In some implementations, the thread pool executor can use the pre-defined ThreadPoolExecutor. ThreadPoolExecutor is a thread pool implementation class provided by Facebook's open-source Folly library, used for efficient management and scheduling of thread resources to achieve parallel execution of distributed computing tasks.

[0043] The implementation of the Folly coroutine library:

[0044] Folly provides a coroutine library that can be used in conjunction with thread pools to achieve asynchronous execution and scheduling of distributed computing tasks. A coroutine is a lightweight thread with lower switching costs, allowing multiple tasks to be executed concurrently within a single thread. In this mechanism, the asynchronous execution of coroutines can fully utilize computing resources, improving system throughput and responsiveness. For example, Fiber, a coroutine implementation provided by the Folly library, can efficiently implement coroutines, avoiding the overhead of thread switching and improving program performance.

[0045] Achieving Overlapping Computation and Communication: By using Brpc for network communication, Foly's thread pool and coroutine library can achieve overlap between computation and communication. While one task is performing computation, another task can be started simultaneously for network communication. This hides communication time within computation time, improving system efficiency. For example, when processing large-scale data, data computation can be performed in one coroutine, while data transmission is conducted via Brpc in another coroutine, achieving parallel execution of computation and communication.

[0046] Therefore, the combined use of Folly's thread pool, coroutine library, and Brpc provides efficient asynchronous execution and scheduling capabilities for graph computing external memory expansion technology based on the domestic environment, realizes the overlap of computation and communication, makes full use of computing resources, and improves system performance and efficiency.

[0047] This application shards graph computation tasks by real-time sensing of system resources and allocates resources such as thread pools to the obtained node tasks, avoiding delays caused by cluster resource constraints or unreasonable allocation strategies. By executing the graph data lookup process through synchronous and asynchronous processing, targeted computations for different node tasks can be achieved, ensuring data consistency while avoiding thread idleness. The thread pool simultaneously retrieves data from other nodes (for data exchange) and the caching module (for node-cached data), reducing network transmission overhead. When the cache is not hit, data is automatically loaded from a preset database and written to the caching module, improving the execution efficiency of subsequent tasks and thus enhancing the overall efficiency of graph computation in large-scale data processing.

[0048] According to some embodiments, in the process of partitioning a graph computation task to obtain node tasks and corresponding thread pools, so that the thread pools can obtain exchange data from other node tasks and node cache data in the cache module, the specific steps may be as follows: parse task information to determine the task complexity and data volume of the graph computation task; allocate resources according to the task complexity, data volume and real-time system resources to obtain node tasks and thread pools, so that the thread pools can obtain exchange data and node cache data.

[0049] In this application, the task information may include the amount of data for the task, as well as the intuitive task complexity.

[0050] In some implementations, resource allocation strategies can be pre-set, including but not limited to:

[0051] (1) Allocate thread resources according to complexity:

[0052] For complex tasks (such as full-map PageRank): set the thread pool size to 1.5 times the number of CPU cores (to make full use of multi-cores); for simple tasks (such as local SSSP): set the thread pool size to 0.5 times the number of CPU cores (to avoid thread contention).

[0053] (2) Allocate storage resources according to data volume:

[0054] Data volume ≤ 50% of memory: all data is cached in DRAM; 50% < data volume ≤ 80%: hot data is stored in DRAM and cold data is stored in NVM (distinguished by LRU strategy); data volume > 80%: external storage expansion mechanism is triggered, and some data is persisted to SSD.

[0055] (3) Adjust dynamically according to system resource status:

[0056] When CPU utilization is >90%, new task submissions are suspended, and high-priority tasks are processed first. When network bandwidth is insufficient, communication-intensive tasks and computing tasks are scheduled to avoid peak times (e.g., communication operations are executed asynchronously through coroutines).

[0057] Through the three processes described above, resource allocation can be achieved, obtaining node tasks, thread pools, and other related resources for data acquisition and retrieval.

[0058] This application dynamically allocates computing resources based on task complexity and data volume, combined with real-time system resources (such as CPU, memory, and bandwidth), to avoid resource waste or overload. By rationally allocating node tasks and thread pools, it prevents some nodes or threads from being overloaded while others remain idle, thereby improving the overall system throughput and response speed. The thread pool can simultaneously obtain exchanged data (i.e., intermediate results or shared data) from other node tasks and node cached data (i.e., locally cached intermediate results or historical data) from the caching module, reducing data transmission latency and avoiding redundant calculations, thus significantly improving data processing efficiency.

[0059] According to some embodiments, in order to determine the task complexity of a graph computing task, the following steps can be taken: extract task description information and data volume from the task information, wherein the task description information includes the graph computing algorithm type, iteration parameters, and parallelism requirements; and match a preset complexity level according to the task description information to determine the task complexity.

[0060] In this application, task information may include the amount of data for the task, as well as relevant metrics for determining complexity. These metrics include the type of graph computation algorithm, iteration parameters, and parallelism requirements. In some implementations, the mathematical characteristics of the graph computation algorithm can be the core, and the corresponding relevant metrics may include the number of iterations (e.g., 20 iterations in the test) and convergence speed of the PageRank (PageRank Algorithm) algorithm; and the shortest path search range (e.g., full graph traversal or local subgraph) and number of starting points (averaging 10 vertices) of the SSSP (Single-Source Shortest Path) algorithm. Parallelism requirements can be used to represent the task's concurrency requirements for threads / coroutines. For example, computationally intensive tasks (e.g., iterative calculation of vertex importance) require high-parallelism thread pools; sparse tasks (e.g., random access to edge data) have lower concurrency requirements.

[0061] In some implementations, task description information and data volume can be extracted from task information. A mapping relationship between complexity levels and information is pre-defined, and then the complexity level corresponding to the task description information of each node is matched in the mapping relationship, and this complexity level is used to represent the task complexity.

[0062] This application comprehensively characterizes task features by extracting the graph computation algorithm type, iteration parameters, and parallelism requirements, avoiding misjudgments of complexity caused by a single indicator (such as data volume). Task characteristics are matched with preset complexity levels (such as low, medium, and high), enabling rapid and accurate quantification of task complexity.

[0063] According to some embodiments, when determining the complexity level, the execution status of historical tasks can also be considered. Specifically, this can be done by: searching for historical tasks to identify historical subtasks that match the graph computation task; and obtaining historical statistics of historical subtasks to match the complexity level based on the historical statistics.

[0064] In this application, previously executed historical tasks can be recorded separately. If there are historical subtasks in the historical tasks that match the current graph computation task type, historical statistics of the historical subtasks can be obtained, including task execution time, resource utilization, etc., and then graph computation tasks of this type that meet the preset criteria can be automatically classified and their complexity levels can be further calibrated.

[0065] Traditional complexity assessments may rely solely on task description information, but in actual execution, factors such as data distribution and computing environment can affect complexity. By incorporating historical statistical data, the complexity level can be dynamically calibrated, reducing the deviation between the theoretical model and actual execution.

[0066] According to some embodiments, the caching module includes memory and non-volatile memory. The conditions for whether node cache data is obtained can be varied, specifically: controlling the thread pool to search memory; if memory does not contain node cache data, controlling the thread pool to search non-volatile memory asynchronously; if non-volatile memory does not contain node cache data, or memory contains expired node cache data, determining that node cache data has not been obtained; and if memory contains non-expired node cache data, or non-volatile memory contains node cache data, determining that node cache data has been obtained.

[0067] In some implementations, memory can be used to store a portion of frequently used data for fast access, while non-volatile memory can be used to store a larger amount of data and can receive data evicted from memory for persistent storage.

[0068] First, follow the above process to determine the thread pools corresponding to the graph computation tasks and node tasks, and then control the thread pools to access memory. Figure 1 The DRAM cache is shown. If the memory does not contain the cached data, the thread pool can be controlled to asynchronously search for the node cached data in non-volatile memory. The non-volatile memory corresponds to... Figure 1 The NVM cache is shown.

[0069] If the non-volatile memory also does not contain node cache data, it can be determined that the node cache data has not been obtained. Additionally, if the memory contains node cache data but it is indeed expired ground cache data, it can also be considered that the node cache data has not been obtained.

[0070] In some implementation manners, the methods for determining whether data has expired include:

[0071] A: Storage and verification mechanism for expiration time:

[0072] Cache item metadata marking: When each cache key-value pair is stored in the DRAM cache, it will be accompanied by an expiration timestamp (such as expire_time), which is determined by the parameters of the expiration-based cleaning policy (such as the expiration time threshold set by the user). For example, if the expiration time of the cache data is set to 1 hour, then expire_time is the storage time + 1 hour.

[0073] Real-time time comparison: When the function findFastImpl is called, the API (Application Programming Interface) will obtain the current system time and compare it with the expire_time of the cache key. If the current time ≥ expire_time, it is determined that the cache key has expired, mark the data and return "not found"; if the current time < expire_time, it is determined that the data is valid and the cache value is normally returned. In this application, findFastImpl is the core internal implementation function in the CacheLib library for quickly finding cache items and verifying their validity.

[0074] B: Basis for setting the expiration time

[0075] User-configured aging parameters: The expiration judgment of findFastImpl depends on the aging-based cleaning policy configuration of CacheLib. For example, the survival time of cache data is set through API parameters, such as setExpireTime(3600 seconds), and this parameter directly determines the calculation method of expire_time; different types of data can be configured with different expiration times (such as hot vertex data having a longer survival time and cold data having a shorter survival time).

[0076] Synergy with eviction policies: The setting of expiration time complements eviction policies such as LRU (Least Recently Used) / LFU (Least Frequently Used). Specifically, time-based cleanup proactively evicts expired data based on time dimension to avoid invalid data occupying memory; LRU and other policies evict cold data based on access frequency. Together, they ensure efficient use of cache space (e.g., "data that has not been accessed for a long time and has low importance is evicted first").

[0077] C: Process for handling expired data

[0078] The tagging and eviction mechanism works in tandem: When findFastImpl determines that a cache key has expired, it performs the following actions: It marks the data as "expired" in the DRAM cache, but does not immediately physically delete it; instead, it waits for subsequent eviction mechanisms to handle it; it returns a "not found" result (corresponding to...). Figure 1 The process of searching for graph data in the NVM (①) triggers the subsequent process of searching for data from the NVM.

[0079] Cache consistency guarantee: After expired data is marked in DRAM, if there is a subsequent write operation (such as data update), the expired data will be replaced or deleted first to ensure the consistency between the cache and the data source, such as "update strategy: update the database first, and then invalidate the cache".

[0080] If the memory contains unexpired node cache data, or if the non-volatile memory contains node cache data, then it can be determined that the node cache data can be retrieved.

[0081] This application uses primary memory as a level-one cache with nanosecond-level access latency. The thread pool prioritizes memory lookups to quickly retrieve frequently accessed data. Non-volatile memory can expand the cache capacity, serving as a level-two cache to store cold data or historical versions. When memory is missed, asynchronous access to the NVM avoids blocking the main thread, while leveraging the NVM's large capacity to cache more data and reduce disk I / O overhead. When memory is missed, the thread pool asynchronously searches the NVM, allowing the main thread to continue processing other tasks (such as computations on other nodes), avoiding thread idleness caused by synchronous waiting and improving system parallelism. Asynchronous access allows computation tasks and data loading to execute in parallel. For example, while thread A is searching for data in the NVM, thread B can utilize the CPU cache to perform other computations, thereby hiding I / O latency and improving overall throughput.

[0082] According to some embodiments, task information represents a first cache allocation space in memory and a second cache allocation space in non-volatile memory. In the event that no node cache data is found, the write process of asynchronously searching for a preset database may specifically include: searching for a preset database when no node cache data is found, writing the found node cache data into the first cache allocation space in memory for caching, and writing it into the second cache allocation space in non-volatile memory for persistence, so that the thread pool can asynchronously return the search results and re-acquire the data synchronously.

[0083] In some implementations, the task information also includes cache space allocation information, i.e. the location of different allocated spaces, wherein the first cache allocation space is used to represent the space available for the node task in memory, and the second cache allocation space is used to represent the space available for the node task in non-volatile memory.

[0084] If no node cache data is found, the system can asynchronously search the preset database and write the found node cache data into the allocated first cache space, achieving in-memory caching for subsequent thread pool calls. Then, it can write the data into the allocated second cache space for persistent storage in non-volatile memory. When the thread pool fails to find node cache data, it asynchronously returns the search result. Based on this result, the graph computing server can search the preset database and rewrite the data. The thread pool can then retrieve the node cache data synchronously after the rewrite.

[0085] This application's caching mechanism persists graph data loaded from the underlying graph database (i.e., the default database), thus avoiding the need to load the entire dataset from the graph database each time and handling datasets exceeding server memory capacity. By simultaneously writing to memory and NVM, a balance is struck between data reliability and access efficiency. Memory provides low-latency access, while NVM ensures no data loss. After completing a database lookup, the thread pool immediately returns the lookup result asynchronously. The main thread can continue executing other tasks without waiting for the data to be actually written to memory and NVM, avoiding thread idleness caused by synchronous writes and improving system parallelism. After asynchronous feedback, the thread pool re-retrieves the data synchronously (i.e., reads the written data from memory or NVM). Synchronous retrieval ensures that the data has been completely written to the caching module, avoiding data inconsistency issues caused by asynchronous write latency.

[0086] According to some embodiments, when a graph computing task includes a data update process, data updates can be performed relative to a preset database. Specifically, this can involve: obtaining data to be updated from the graph computing task; updating the preset database based on the data to be updated; and invalidating the corresponding data stored in the cache module upon completion of the update.

[0087] In this application, both the cache module and the preset database need to be updated when data updates are required. In some implementations, the data to be updated is first determined from the graph computation task. In the preset database, the data to be updated is used to overwrite the original data in the database, and then the original data in the storage module is invalidated. When the data is accessed again, it is read from the database again and the cache module is updated. During this process of reading data and updating the cache, a check operation is performed before writing the newly read data to the cache module to confirm the validity of the data, whether it is duplicated, etc., to avoid writing erroneous data into the cache.

[0088] During the overwrite process, the data interface, such as the insertOrReplace in the API, encapsulates the atomic operation of "marking deletion + writing new data" to ensure that the data will not be in an intermediate state during the replacement process; the original data is marked for deletion in NVM instead of being deleted immediately, and is subsequently cleaned up in batches by a background thread to reduce I / O overhead ("the data in NVM is marked for deletion, and is subsequently deleted asynchronously by calling a preset function").

[0089] Because it's necessary to ensure that the written data meets cache requirements, such as checking for duplicate data, if duplicate data exists, `insertOrReplace` can be called to directly replace the existing data. In this case, a check is needed first to determine if the data is duplicated. This process is called the `check` operation. Besides the data update process, there may also be a data write process, in which case a `check` operation is also required.

[0090] This application immediately invalidates the corresponding data in the cache module after updating the preset database, ensuring that subsequent graph calculation tasks can only read the latest data. When other tasks query this list, a reload will be triggered, avoiding calculation errors caused by outdated cache. By proactively invalidating the cache, it avoids a large number of requests directly penetrating to the database due to outdated cache, or a surge in database pressure caused by concentrated cache expiration.

[0091] According to other embodiments, this application can also evict data in the cache module.

[0092] In the data eviction process, considering the limited memory capacity and the differences in data access frequency, the caching mechanism prioritizes evicting data that has not been accessed for a long time and has relatively low importance from memory and transferring it to other storage levels.

[0093] The methods for determining whether an area has not been accessed for a long time include:

[0094] A. Time-threshold-based expiration determination: A time-sensitive cleanup strategy is used to set a fixed expiration time (e.g., minutes, hours) for cached data. If data is not accessed within this time threshold, it is determined to be "unaccessed for a long time". For example, if the expiration time is set to 1 hour, data that has not been read for more than 1 hour meets this condition.

[0095] B. Frequency-based determination based on access history: When using the LRU strategy, data that has not been accessed for a long time is considered "not accessed for a long time" by recording the last access time of the data. For example, when memory space is insufficient, cached data that has not been accessed for the longest time is evicted first.

[0096] Methods for determining importance that are relatively low include:

[0097] A. Quantitative standard based on access frequency: By statistically analyzing the access frequency using the LFU (Low-Frequency Requirement) strategy, data with an access frequency below a set threshold is considered "relatively low in importance." For example, if a certain type of data is accessed far less frequently than other data within a unit of time, it is considered to have low importance.

[0098] B. Priority Ranking Based on Elimination Strategy: In a number-threshold-based cleanup strategy (such as FIFO, First In First Out), the "importance" of data is related to the access sequence or business attributes. For example, newly written data has a higher priority than old data, and old data is considered less important due to its "later access sequence" and is eliminated first. Here, "importance" is determined by default based on data access frequency as the core standard, without considering the data's business attributes (such as whether it is core business data), and is quantified only through technical indicators such as access frequency and time intervals.

[0099] The caching mechanism in this application filters and eliminates data based on two criteria: "time dimension (duration of non-access)" and "frequency dimension (access importance)". Data will be prioritized for elimination if both criteria are met simultaneously (e.g., long period of non-access and low access frequency).

[0100] The CacheLib used in this application supports custom eviction policies (such as hybrid LRU+LFU), so the specific thresholds for "long time" and "importance" can be adjusted through parameter configuration to adapt to the cache management needs of different business scenarios.

[0101] In some implementations, when data meets the eviction policy requirements, there may be two paths: direct eviction from CacheLib or eviction from memory into the NVM. Before deciding which eviction path to take, a check operation may be performed. For example, the NVM's space availability and the data's characteristics (whether it's cold data, whether it has potential access value) may be checked to determine which path to choose for eviction.

[0102] According to other embodiments, the graph computation method of this application can be named the PageRank algorithm. The PageRank algorithm and other algorithms can be tested using a domestically developed platform. Reference can be made to Table 1, which shows the test results of the PageRank algorithm's graph system running in memory on an Intel platform; Table 2, which shows the test results of the SSSP algorithm running in memory on an Intel platform; Table 3, which shows the test results of the PageRank algorithm's graph system running in external memory on an Intel platform; Table 4, which shows the test results of the SSSP algorithm running in external memory on an Intel platform; Table 5, which shows the test results of the PageRank algorithm's graph system running in memory on a domestically developed Phytium platform; and Table 6, which shows the test results of the SSSP algorithm running in memory on a domestically developed Phytium platform. The test results are as follows: Table 7 shows the test results of the PageRank algorithm graph system running the algorithm using external storage on the domestic environment Phytium; Table 8 shows the test results of the SSSP algorithm running the algorithm using external storage on the domestic environment Phytium; Table 9 shows the test results of the PageRank algorithm graph system running the algorithm using external storage on the domestic environment Kunpeng; Table 10 shows the test results of the SSSP algorithm running the algorithm using external storage on the domestic environment Kunpeng; Table 11 shows the test results of the PageRank algorithm graph system running the algorithm using external storage on the domestic environment Kunpeng; and Table 12 shows the test results of the SSSP algorithm running the algorithm using external storage on the domestic environment Kunpeng.

[0103] The PageRank algorithm terminates after 20 iterations, with all vertices active in each iteration. The SSSP algorithm selects 10 vertices as the starting points for each iteration, and the final result is the average. (The starting point numbers are 0, 1, 3, 5, 7, 9, 11, 13, 15, 17, and 43607; each vertex is in the largest connected component of the entire graph). Since GraphX ​​has not been fully adapted for the domestic environment, a three-node comparative test was conducted on the Intel platform to verify the effectiveness of the graph computing external memory expansion technology based on the domestic environment. The test results are shown in Tables 1 and 2. It can be seen that the performance of the optimized graph system far exceeds that of GraphX. From the test results in Table 3-12, its performance is still superior to GraphX, thus proving the effectiveness and advancement of this mechanism.

[0104] Table 1

[0105]

[0106] Table 2:

[0107]

[0108] Table 3:

[0109]

[0110] Table 4:

[0111]

[0112] Table 5:

[0113]

[0114] Table 6:

[0115]

[0116] Table 7:

[0117]

[0118] Table 8:

[0119]

[0120] Table 9:

[0121]

[0122] Table 10:

[0123]

[0124] Table 11:

[0125]

[0126] Table 12:

[0127]

[0128] According to some embodiments, the adaptation steps are as follows:

[0129] Installation preparation requires a GCC (GNU Compiler Collection) environment, preferably GCC 10.3.0. Then, system dependencies such as GMP-devel (GNU Multiple Precision Arithmetic Library Development Files) and mpfr (Multiple Precision Floating-Point Reliable Library) need to be installed. After confirming the system libraries are installed, begin compiling third-party libraries. It is important to note that the dependencies listed in each step must be installed sequentially from left to right and top to bottom, as these libraries may depend on each other. For example, when installing the dependency Glog (Google Logging Library), which depends on gtest (Google C++ Testing Framework) and gflags (Google Command Line Flags), ensure that these two libraries are already correctly installed.

[0130] The following describes an apparatus embodiment of this application, which can be used to perform the method embodiment of this application. For details not disclosed in the apparatus embodiment of this application, please refer to the method embodiment of this application.

[0131] Figure 3 A block diagram of a graph computing device provided in an embodiment of this application. Figure 3 As shown, the computing device 300 includes a resource allocation module 301, a data acquisition module 302, and a computing module 303.

[0132] The resource allocation module 301 is used to determine the node task and the thread pool corresponding to the graph computing task based on the real-time system resources and the task information of the graph computing task, so that the thread pool can obtain the exchange data of other node tasks and the node cache data in the cache module.

[0133] The data acquisition module 302 is used to asynchronously search the preset database when no node cache data is acquired, and write the found node cache data into the cache module so that the thread pool can reacquire it synchronously.

[0134] The calculation module 303 is used to perform multi-threaded synchronous calculations based on the node cache data and exchanged data when the node cache data is obtained, so as to determine the graph calculation result.

[0135] Optionally, the resource allocation module 301 is specifically used for:

[0136] Analyze the task information to determine the task complexity and data volume of the graph computation task;

[0137] Resource allocation is performed based on task complexity, data volume, and real-time system resources to obtain node tasks and thread pools, enabling the thread pools to acquire exchanged data and node cached data.

[0138] Optionally, when parsing task information to determine the task complexity and data volume of the graph computation task, the resource allocation module 301 is specifically used for:

[0139] Extract task description information and data volume from the task information. The task description information includes the graph computation algorithm type, iteration parameters, and parallelism requirements.

[0140] The task complexity is determined by matching the preset complexity level with the task description information.

[0141] Optionally, the graph computing device 300 further includes a complexity level determination module 303, used for:

[0142] Search for historical tasks to identify historical subtasks that match the graph computation task.

[0143] Obtain historical statistics for historical subtasks to match complexity levels based on historical statistics.

[0144] Optionally, the cache module includes memory and non-volatile memory; the graph computing device 300 also includes an acquisition and determination module 305, used for:

[0145] Control the thread pool's memory access;

[0146] If the memory does not contain node cache data, control the thread pool to search for non-volatile memory asynchronously;

[0147] If the non-volatile memory does not contain node cache data, or if the memory contains expired node cache data, it is determined that node cache data has not been retrieved.

[0148] If the memory contains non-expired node cache data, or if the non-volatile memory contains node cache data, then the node cache data is determined to have been retrieved.

[0149] Optionally, the task information represents the first cache allocation space in memory and the second cache allocation space in non-volatile memory; the data acquisition module 302 is specifically used for:

[0150] If no node cache data is found, the preset database is searched asynchronously. The found node cache data is then cached in the first cache allocation space in memory and persisted in the second cache allocation space in non-volatile memory. This allows the thread pool to provide the search results asynchronously and retrieve the data again synchronously.

[0151] Optionally, when the graph computing task includes a data update process, the graph computing device 300 further includes an update module 306, used for:

[0152] Obtain the data to be updated from the graph computation task;

[0153] Based on the data to be updated, the preset database is updated, and the corresponding data stored in the cache module is invalidated when the update is complete.

[0154] The device performs functions similar to those described above; other functions are described in the preceding descriptions and will not be repeated here.

[0155] Figure 4 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application, such as... Figure 4 As shown, the electronic device 400 of this embodiment may include a memory 401 and a processor 402.

[0156] The memory 401 stores a computer program, which, when executed by the processor 402, causes the processor 402 to perform the method described in the above embodiment.

[0157] The processor 402 and the memory 401 are connected, for example, via a bus.

[0158] Optionally, the electronic device 400 may also include a transceiver. It should be noted that in practical applications, the transceiver is not limited to one, and the structure of the electronic device 400 does not constitute a limitation on the embodiments of this application.

[0159] Processor 402 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It may implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 402 may also be a combination that implements computational functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0160] A bus can include a pathway for transmitting information between the aforementioned components. The bus can be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, only one thick line is used in the diagram, but this does not imply that there is only one bus or one type of bus.

[0161] The memory 401 may be a ROM (Read Only Memory) or other type of static storage device capable of storing static information and instructions, RAM (Random Access Memory) or other type of dynamic storage device capable of storing information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read Only Memory) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto.

[0162] The memory 401 is used to store application code that executes the solution of this application, and its execution is controlled by the processor 402. The processor 402 is used to execute the application code stored in the memory 401 to implement the content shown in the foregoing method embodiments.

[0163] Electronic devices include, but are not limited to: mobile terminals such as mobile phones, laptops, digital radio receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), and in-vehicle terminals (such as in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Servers can also be included. Figure 4 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0164] The electronic device in this embodiment can be used to execute the method of any of the above embodiments, and its implementation principle and technical effect are similar, so they will not be described again here.

[0165] This application also provides a non-transitory computer-readable storage medium storing computer-readable instructions thereon, which, when executed by a processor, cause the processor to perform the method as described in the above embodiments.

[0166] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a non-transitory computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0167] The embodiments of this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only for the purpose of helping to understand the method and core ideas of this application. Furthermore, any changes or modifications made by those skilled in the art based on the ideas of this application, and on the specific implementation methods and application scope of this application, are all within the scope of protection of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A graph calculation method, characterized in that, include: Based on real-time system resources and graph computing task information, determine the node task corresponding to the graph computing task and the thread pool corresponding to the node task, so that the thread pool can obtain the exchange data of other node tasks and the node cache data in the cache module. If the node cache data is not obtained, the preset database is searched asynchronously, and the found node cache data is written to the cache module so that the thread pool can retrieve it again synchronously. If the node cache data is obtained, multi-threaded synchronous calculation is performed based on the node cache data and the exchange data to determine the graph calculation result; The cache module includes memory and non-volatile storage; it also includes: Control the thread pool to search the memory; If the memory does not contain the node cache data, the thread pool is controlled to search for the non-volatile memory asynchronously. If the non-volatile memory does not contain the node cache data, or if the memory contains expired node cache data, it is determined that the node cache data has not been obtained. If the memory contains the node cache data that has not expired, or if the non-volatile memory contains the node cache data, then it is determined that the node cache data has been obtained. Wherein, the task information represents the first cache allocation space in the memory and the second cache allocation space in the non-volatile memory; the step of asynchronously searching a preset database and writing the found node cache data into the cache module when the node cache data is not obtained, so that the thread pool can re-acquire the data synchronously, includes: If the node cache data is not obtained, the preset database is searched asynchronously, and the found node cache data is written into the first cache allocation space of the memory for caching, and written into the second cache allocation space of the non-volatile memory for persistence, so that the thread pool can return the search result asynchronously and retrieve it again synchronously.

2. The method according to claim 1, characterized in that, The step of determining the node task corresponding to the graph computing task and the thread pool corresponding to the node task based on real-time system resources and task information of the graph computing task, so that the thread pool can obtain exchange data from other node tasks and node cache data in the cache module, includes: The task information is analyzed to determine the task complexity and data volume of the graph computation task; Resource allocation is performed based on the task complexity, the data volume, and the real-time system resources to obtain the node task and the thread pool, so that the thread pool can acquire the exchanged data and the node cached data.

3. The method according to claim 2, characterized in that, The process of parsing the task information to determine the task complexity and data volume of the graph computation task includes: Extract task description information and data volume from the task information, wherein the task description information includes graph computation algorithm type, iteration parameters and parallelism requirements; The task complexity is determined by matching a preset complexity level with the task description information.

4. The method according to claim 3, characterized in that, Also includes: Search for historical tasks to identify historical subtasks that match the graph computation task from the historical tasks; Obtain historical statistics data for the historical subtasks to match the complexity level based on the historical statistics data.

5. The method according to claim 1, characterized in that, When the graph computation task includes a data update process, the method further includes: Obtain the data to be updated from the graph computation task; Based on the data to be updated, the preset database is updated, and the corresponding data stored in the cache module is invalidated when the update is completed.

6. A graph computing device, characterized in that, include: The resource allocation module is used to determine the node task corresponding to the graph computing task and the thread pool corresponding to the node task based on the real-time system resources and the task information of the graph computing task, so that the thread pool can obtain the exchange data of other node tasks and the node cache data in the cache module. The data acquisition module is used to asynchronously search a preset database when the node cache data is not acquired, and write the found node cache data into the cache module so that the thread pool can reacquire the data synchronously. The calculation module is used to perform multi-threaded synchronous calculations based on the node cache data and the exchange data when the node cache data is obtained, so as to determine the graph calculation result; The cache module includes memory and non-volatile storage; it also includes an acquisition and determination module for: Control the thread pool to search the memory; If the memory does not contain the node cache data, the thread pool is controlled to search for the non-volatile memory asynchronously. If the non-volatile memory does not contain the node cache data, or if the memory contains expired node cache data, it is determined that the node cache data has not been obtained. If the memory contains the node cache data that has not expired, or if the non-volatile memory contains the node cache data, then it is determined that the node cache data has been obtained. The task information represents the first cache allocation space in the memory and the second cache allocation space in the non-volatile memory; the data acquisition module is specifically used for: If the node cache data is not obtained, the preset database is searched asynchronously, and the found node cache data is written into the first cache allocation space of the memory for caching, and written into the second cache allocation space of the non-volatile memory for persistence, so that the thread pool can return the search result asynchronously and retrieve it again synchronously.

7. An electronic device, characterized in that, include: processor; A memory storing a computer program that, when executed by the processor, causes the processor to perform the method as described in any one of claims 1-5.

8. A non-transitory computer-readable storage medium, characterized in that, It stores computer-readable instructions that, when executed by a processor, cause the processor to perform the method as described in any one of claims 1-5.

Images