Large-scale graph data real-time calculation method and system based on big data

Abstract

The invention relates to a large-scale graph data real-time calculation method and system based on big data. The method comprises the following steps: converting original graph data into a compressed sparse row format, filtering redundancy, generating a plurality of independent sub-tasks based on edge division, and injecting the independent sub-tasks into a shared task pool of a CPU (Central Processing Unit) main memory; the CPU thread and the FPGA kernel serve as peer-to-peer computing units to dynamically and concurrently pull tasks from the pool, self-adaptive load balancing is achieved, and associated data guaranteeing computing integrity is transmitted to the FPGA unit; the CPU unit executes a kernel function based on SIMD and dynamic algorithm selection optimization, and the FPGA unit executes a customized kernel function based on parallel pipeline binary search; and finally, aggregating all local results to generate a final calculation result. Through software and hardware cooperation and dynamic scheduling, the throughput rate, the energy efficiency and the real-time performance of large-scale graph data processing are remarkably improved.

Description

Technical Field

[0001] This application relates to the field of graph data technology, and in particular to a method and system for real-time computation of large-scale graph data based on big data. Background Technology

[0002] With the rapid development of fields such as social network analytics, financial risk control, knowledge graph construction, and the Internet of Things, the complex relationships between data are often modeled and stored using graph structures, leading to an explosive growth in the scale of graph data. Real-time analysis of this large-scale graph data, such as real-time community discovery, fraud transaction tracking, or dynamic recommendations, has become a key requirement for supporting intelligent decision-making.

[0003] Existing general-purpose parallel acceleration solutions based on GPUs typically load the entire graph data into the GPU's global memory, leveraging the GPU's massive computing cores (CUDA Cores) to perform large-scale parallel processing of graph computation tasks. First, the graph data is preprocessed, such as by reordering or partitioning vertices to optimize memory access patterns. Then, thousands of threads are launched on the GPU to perform computations in an "edge-centric" or "vertex-centric" parallel mode. During parallel execution, to alleviate the problem of uneven load distribution among threads, a significant amount of time is usually invested in complex static task partitioning and load estimation during the preprocessing stage to ensure that the computational workload of each thread is roughly balanced during computation.

[0004] The aforementioned technical solutions heavily rely on static task partitioning and load estimation during the preprocessing stage. However, large-scale graph data is often highly irregular, and static partitioning struggles to accurately predict the actual overhead of each task during runtime. This leads to some threads being overloaded while others become idle early during computation, resulting in uneven load distribution and hindering the full utilization of computing resources.

[0005] Based on this, this application provides a method and system for real-time computation of large-scale graph data based on big data. Summary of the Invention

[0006] To address the issue of heavy reliance on static task partitioning and load estimation during the preprocessing stage, this application provides a real-time computation method and system for large-scale graph data based on big data. However, large-scale graph data is often highly irregular, and static partitioning struggles to accurately predict the actual overhead of each task during runtime. This leads to some threads being overloaded while others are idle, resulting in uneven load distribution and hindering the full utilization of computing resources.

[0007] Firstly, this application provides a real-time computation method for large-scale graph data based on big data, employing the following technical solution: including: The system receives raw large-scale graph data, converts it into a compressed sparse row format, and filters out redundant data to obtain optimized graph data. Based on the optimized graph data, it adopts an edge-based partitioning strategy to stream-divide the computation task into multiple independent subtasks, and generates a task descriptor containing its data range index for each subtask. All the task descriptors are injected into a shared task pool located in the CPU main memory. The shared task pool is configured as a circular queue structure that supports concurrent access by multiple computing units. One or more computing threads on the CPU and one or more computing cores on the FPGA are used as computing units to concurrently pull the task descriptors to be processed from the shared task pool. For the subtasks pulled by the FPGA computing units, the graph data segment corresponding to the subtask and the associated data necessary to ensure computing integrity are transmitted to the onboard memory of the FPGA via the bus. The CPU computing unit accesses the corresponding graph data segment from main memory based on the retrieved task descriptor, executes the first graph computing kernel function optimized for the CPU architecture, and obtains the first local result; the FPGA computing unit accesses data from its onboard and on-chip memory, executes the second graph computing kernel function optimized for the FPGA pipeline architecture, and the second graph computing kernel function accelerates the calculation through deep pipeline and parallel processing to obtain the second local result. The second local results generated by each FPGA computing unit are sent back to the CPU main memory. The CPU then aggregates the first local results and the second local results from all computing units to generate the final calculation result of the large-scale graph data.

[0008] Preferably, the process involves receiving the original large-scale graph data, converting it into a compressed sparse row format, and filtering out redundant data to obtain optimized graph data. Based on the optimized graph data, an edge-based partitioning strategy is used to stream-divide the computational task into multiple independent subtasks. A task descriptor containing its data range index is generated for each subtask. All the task descriptors are injected into a shared task pool located in the CPU main memory. The shared task pool is configured as a circular queue structure supporting concurrent access by multiple computational units, including: Receive raw large-scale graph data and convert it into a compressed sparse row format. The compressed sparse row format includes a vertex offset array and a neighbor vertex identifier array to optimize storage and adjacency query efficiency. An iterative filtering algorithm is performed on the compressed sparse line format graph data to remove redundant vertices and their associated edges that do not meet the preset calculation conditions, and to generate optimized graph data with reduced size. Based on the neighbor vertex identifier array of the optimized graph data, an edge-based partitioning strategy is adopted to divide the entire computation task into multiple subtasks. Each subtask corresponds to a continuous and non-overlapping index interval in the array. For each subtask obtained by the division, a corresponding task descriptor is generated. The task descriptor at least records the start index, end index and range of vertices involved in the corresponding subtask in the neighbor vertex identifier array. The generated task descriptors are stored sequentially into the shared task pool, which is constructed as a circular queue structure that supports concurrent access and maintains head and tail pointers to manage task status.

[0009] Preferably, the step of performing an iterative filtering algorithm on the compressed sparse row format graph data to remove redundant vertices and their associated edges that do not meet the preset calculation conditions, and generating optimized graph data with reduced size, includes: Traverse all vertices in the compressed sparse row format graph data, and add vertices that do not meet the preset calculation conditions to a deletion queue. The preset calculation conditions include at least the degree of the vertex not being less than a preset threshold. When the queue to be deleted is not empty, an iterative process is executed. Each iteration includes: taking a current vertex to be deleted from the queue to be deleted; removing the current vertex to be deleted and all its associated edges from the graph data structure; updating the degree of each neighboring vertex of the current vertex to be deleted; if the degree of the neighboring vertex after the update does not meet the preset calculation condition, then adding the corresponding neighboring vertex to the queue to be deleted. Iteration Termination and Output: Repeat the iteration process until the queue to be deleted is empty, and use the filtered graph data as the optimized graph data after scaling down.

[0010] Preferably, the step of using one or more computing threads on the CPU and one or more computing cores on the FPGA as computing units to concurrently pull the task descriptors to be processed from the shared task pool, and for the subtasks pulled by the FPGA computing units, transmitting the graph data segment corresponding to the subtask and its associated data necessary to ensure computational integrity to the onboard memory of the FPGA via the bus, includes: This enables each computing thread on the CPU and each computing core on the FPGA to enter a ready state and concurrently access the shared task pool located in the CPU main memory. Each computing unit performs a pull operation to acquire the access lock of the shared task pool, reads the task descriptor currently indicated by the head pointer as the subtask to be processed, then updates the head pointer to point to the next task descriptor to be processed, and releases the access lock. When the computing unit performing the pull operation is an FPGA computing kernel, after obtaining the task descriptor of the subtask to be processed, the corresponding graph data segment is located from the optimized graph data based on the data range index contained in the task descriptor. At the same time, according to the graph topology, the associated data related to the graph data segment and the necessary associated data to ensure the semantic integrity of the graph computing kernel function executed on the corresponding subtask are determined and read. The graph data segment and associated data are transmitted via the system bus to the onboard memory corresponding to the FPGA computing core that initiated the pull, thus completing the data deployment before computation.

[0011] Preferably, the CPU computing unit accesses the corresponding graph data segment from main memory based on the retrieved task descriptor, executes a first graph computing kernel function optimized for the CPU architecture, and obtains a first local result, including: Based on the data range index recorded in the task descriptor retrieved by the CPU computing unit, the corresponding graph data segment is located and loaded into the CPU cache from the optimized graph data in main memory; For the computational units to be processed in the graph data segment, obtain the corresponding set of data to be compared; based on the size characteristics of the set of data to be compared, dynamically select to execute the merge algorithm or the binary search algorithm for the core computational operation, and use single instruction multiple data stream instructions to vectorize and accelerate the execution of the selected algorithm; After processing all computational units within the graph data segment, the computational results are summarized to generate the first local result.

[0012] Preferably, the FPGA computing unit accesses data from its onboard and on-chip memory, executes a second graph computation kernel function optimized for the FPGA pipeline architecture, and the second graph computation kernel function accelerates computation through deep pipeline and parallel processing to obtain a second local result, including: Based on the FPGA computing unit, the graph data segment and associated data are prefetched from the corresponding onboard memory to the on-chip cache of the FPGA in a burst transfer manner; Based on the data in the cache, multiple parallel binary search pipeline processing units are started. Each processing unit performs the following operations according to the pipeline cycle: read the query data to be processed from the cache; perform binary search in the target data set read from on-chip memory to perform set intersection calculation or pattern matching calculation to obtain the calculation result; and accumulate the calculation result into the local accumulator corresponding to the corresponding processing unit. After all parallel pipeline processing units have completed the calculation of the allocated data, the results of the local accumulators of each processing unit are reduced to generate the second local result.

[0013] Preferably, the step of sending the second local results generated by each FPGA computing unit back to the CPU main memory, and then having the CPU aggregate the first local results and the second local results from all computing units to generate the final calculation result of the large-scale graph data, includes: Based on the second local results generated by each FPGA computing unit, the respective second local results are directly written into the pre-allocated, unified result buffer in the CPU main memory via the system bus; After the result feedback is completed or the preset synchronization point is reached, a main control thread of the CPU is responsible for collecting the second local results fed back by all FPGA computing units from the result buffer, and collecting the first local results generated by each CPU computing thread. The CPU's main control thread performs a global aggregation operation on the collected first local results and second local results. The global aggregation operation includes at least one of the following: accumulation, merging, or reduction operations, to generate the final calculation result of the large-scale graph data.

[0014] Secondly, this application discloses a real-time computing device for large-scale graph data based on big data, which adopts the following technical solution, including: The data processing module receives raw large-scale graph data, converts it into a compressed sparse row format, and filters out redundant data to obtain optimized graph data. Based on the optimized graph data, it adopts an edge-based partitioning strategy to stream-divide the computation task into multiple independent subtasks, generates a task descriptor containing its data range index for each subtask, and injects all the task descriptors into a shared task pool located in the CPU main memory. The shared task pool is configured as a circular queue structure that supports concurrent access by multiple computing units. The task allocation module is used to use one or more computing threads on the CPU side and one or more computing cores on the FPGA side as computing units to concurrently pull the task descriptors to be processed from the shared task pool. For the subtasks pulled by the FPGA computing units, the graph data segment corresponding to the subtask and the associated data necessary to ensure the integrity of the calculation are transmitted to the onboard memory of the FPGA via the bus. The task processing module is used by the CPU computing unit to access the corresponding graph data segment from main memory according to the retrieved task descriptor, execute the first graph computing kernel function optimized for the CPU architecture, and obtain the first local result; the FPGA computing unit accesses data from its onboard and on-chip memory, executes the second graph computing kernel function optimized for the FPGA pipeline architecture, and the second graph computing kernel function accelerates the calculation through deep pipeline and parallel processing to obtain the second local result. The result aggregation module is used to send the second local results generated by each FPGA computing unit back to the CPU main memory. The CPU then aggregates the first local results and the second local results from all computing units to generate the final calculation result of the large-scale graph data.

[0015] Thirdly, this application also provides a control device, the device comprising: It includes a memory and a processor, wherein the memory stores a computer program that can be loaded by the processor and executed, such as the above-described method for real-time computation of large-scale graph data based on big data.

[0016] Fourthly, this application also provides a computer-readable storage medium storing a computer program that can be loaded by a processor and executed as described above regarding the real-time computation method for large-scale graph data based on big data.

[0017] In summary, this application's system converts the original graph data into a compressed sparse row format and performs iterative filtering to remove redundancy, thus obtaining optimized graph data. Subsequently, an edge-based partitioning strategy is adopted to stream-divide the computational task into multiple independent subtasks, each stored as a lightweight descriptor in a shared task pool within the CPU's main memory. Then, based on a dynamic cooperative scheduling mechanism, CPU threads and FPGA cores, as peer computing units, actively and concurrently pull tasks from the shared pool. This pull-based model naturally achieves dynamic load balancing based on the actual processing speed of each unit, realizing runtime adaptive task allocation, greatly improving the utilization of computing resources, and further enhancing real-time processing performance. For tasks pulled by the FPGA, the system transmits their corresponding data segments and associated data necessary to ensure computational integrity to the FPGA's onboard storage via the bus. During the execution phase, the CPU unit executes optimized kernel functions using SIMD instruction sets and dynamic algorithms, while the FPGA unit executes customized hardware kernel functions based on deep pipelines and parallel binary search, fully leveraging the advantages of their respective architectures. Finally, all scattered local results are fed back and aggregated to generate the final computational result. Through a systematic design involving dynamic scheduling, heterogeneous collaboration, and joint hardware and software optimization, this approach effectively reduces the uneven load caused by static partitioning and the low energy efficiency resulting from a single hardware architecture, thus achieving low-latency and high-energy-efficiency real-time processing of large-scale graph data. Attached Figure Description

[0018] Figure 1 This is a flowchart illustrating a real-time computation method for large-scale graph data based on big data.

[0019] Figure 2 This is a structural block diagram of a large-scale graph data real-time computing device based on big data. Detailed Implementation

[0020] The following combination Figure 1 - Figure 2 This application will be described in further detail.

[0021] This application achieves efficient real-time processing of large-scale graph data by constructing a complete closed-loop system from data preprocessing, dynamic task scheduling, heterogeneous parallel execution to result aggregation. To make the objectives, technical solutions, and implementation advantages of this application clearer, the specific implementation methods of this application will be described in detail below with reference to the accompanying drawings and embodiments. The implementation methods will follow the logical framework of the aforementioned technical solutions, and will specifically describe the key steps and collaborative mechanisms of data preprocessing and task generation, dynamic collaborative scheduling, CPU and FPGA heterogeneous computing, and result aggregation. Through the following description, those skilled in the art will be able to fully understand and implement this invention.

[0022] Reference Figure 1 The embodiments of this application include at least steps S10 to S40.

[0023] S10: Receive the original large-scale graph data, convert it into a compressed sparse row format and filter redundant data to obtain optimized graph data; Based on the optimized graph data, adopt an edge-based partitioning strategy to stream-divide the computation task into multiple independent subtasks, and generate a task descriptor containing its data range index for each subtask. Inject all task descriptors into a shared task pool located in the CPU main memory. The shared task pool is configured as a circular queue structure that supports concurrent access by multiple computing units.

[0024] S20 uses one or more computing threads on the CPU and one or more computing cores on the FPGA as computing units to concurrently pull task descriptors to be processed from the shared task pool. For subtasks pulled by the FPGA computing units, the graph data segment corresponding to the subtask and the associated data necessary to ensure computing integrity are transmitted to the onboard memory of the FPGA via the bus.

[0025] S30, the CPU computing unit accesses the corresponding graph data segment from main memory according to the retrieved task descriptor, executes the first graph computing kernel function optimized for the CPU architecture, and obtains the first local result; the FPGA computing unit accesses data from its onboard and on-chip memory, executes the second graph computing kernel function optimized for the FPGA pipeline architecture, and the second graph computing kernel function accelerates the calculation through deep pipeline and parallel processing to obtain the second local result.

[0026] S40 sends the second local results generated by each FPGA computing unit back to the CPU main memory. The CPU then aggregates the first and second local results from all computing units to generate the final calculation result of the large-scale graph data.

[0027] Among them, the dynamic pull mechanism of the shared task pool enables computing units with different processing speeds to obtain different numbers of subtasks, thereby achieving dynamic load balancing among computing units.

[0028] Specifically, the system compresses and filters the original graph data, then decouples the computational tasks into independent subtasks and injects them into a shared task pool through edge-based streaming partitioning. The core innovation lies in the dynamic collaborative scheduling mechanism: CPU threads and FPGA cores, as peer computing units, actively pull tasks from the pool, naturally achieving load balancing through this competition mechanism; task data is distributed and transmitted differentially based on the type of computing unit. During the execution phase, the CPU and FPGA execute kernel functions deeply optimized for their respective hardware characteristics: the CPU uses a SIMD-based dynamic algorithm selection, while the FPGA relies on parallel pipelined custom computing units. Finally, the dispersed results are efficiently aggregated and output. This systematically reduces the occurrence of uneven load, low hardware utilization, and energy efficiency bottlenecks caused by data irregularities in large-scale graph computation, achieving high throughput and low latency real-time processing capabilities through hardware and software co-optimization.

[0029] In some embodiments, step S10 specifically includes the following steps: receiving raw large-scale graph data and uniformly converting it into a compressed sparse row format, the compressed sparse row format including a vertex offset array and a neighbor vertex identifier array to optimize storage and adjacency query efficiency; performing an iterative filtering algorithm on the graph data in the compressed sparse row format to remove redundant vertices and their associated edges that do not meet the preset calculation conditions, generating optimized graph data after scale reduction; based on the neighbor vertex identifier array of the optimized graph data, adopting an edge-based partitioning strategy to divide the entire calculation task into multiple subtasks, each subtask corresponding to a continuous and non-overlapping index interval in the array; generating a corresponding task descriptor for each subtask, the task descriptor at least recording the start index, end index, and vertex range involved in the corresponding subtask in the neighbor vertex identifier array; storing the generated multiple task descriptors in sequence into a shared task pool, the shared task pool being constructed as a circular queue structure supporting concurrent access, and maintaining head and tail pointers to manage task status.

[0030] In performing basic graph calculations such as triangle counting, the problem must first be formally defined. Given an undirected graph... Let V be the set of vertices and E be the set of edges. The goal of the triangle counting problem is to calculate the number of all three-vertex cliques in a graph, as shown in the following formula: ; This represents the total number of triangles in graph G. This definition clarifies the object of computation and provides a precise objective for subsequent algorithm design and optimization.

[0031] Triangle counting is central to calculating network clustering properties. The local clustering coefficient of a vertex quantifies the tightness of connections between its neighboring nodes. For vertex u, its local clustering coefficient... Defined as: ; T(u) represents the number of actual edges between the neighboring nodes of vertex u (i.e., twice the number of triangles containing u). This represents the degree of vertex u. This coefficient is a key indicator for evaluating the local structure of a network.

[0032] Furthermore, the number of triangles is used to calculate the transitivity ratio of the graph, which is an important measure of the overall isomorphism of the network. The transitivity T(G) of a graph is defined as: ; The number of all connected triples in the graph (i.e., two vertices connected by three edges and their common neighbor).

[0033] Specifically, the system unifies the data structure and improves access performance through format conversion, then uses iterative filtering to eliminate invalid data and reduce computational load. Building on this, it decouples the computational task into multiple independently executable subtasks using edge-based continuous partitioning, and represents each task through lightweight index descriptors. Finally, all task descriptors are injected in an ordered manner into a shared task pool based on a circular queue. This lays a crucial foundation for subsequent dynamic scheduling: it improves the parallelism and efficiency of overall processing through data optimization and task decomposition, and decouples computational tasks from specific hardware through the pooled task description structure, thus providing the necessary prerequisite for the CPU and FPGA computing units to pull tasks on demand and achieve adaptive load balancing.

[0034] In some embodiments, in order to dynamically reduce the data size through iterative filtering, step S10 further includes the following steps: traversing all vertices in the compressed sparse row format graph data, adding vertices that do not meet the preset calculation conditions to a deletion queue, the preset calculation conditions including at least the degree of the vertex not being less than a preset threshold; when the deletion queue is not empty, performing an iterative process, each iteration including: taking a current vertex to be deleted from the deletion queue; removing the current vertex to be deleted and all its associated edges from the graph data structure; for each neighboring vertex of the current vertex to be deleted, updating its degree, if the updated neighboring vertex's degree does not meet the preset calculation conditions, then adding the corresponding neighboring vertex to the deletion queue; iteration termination and output: repeating the iterative process until the deletion queue is empty, and using the filtered graph data as the optimized graph data after size reduction.

[0035] Specifically, the system converts graph data into a compressed sparse row format to optimize basic access efficiency, and then initiates a self-driven iterative filtering loop: the algorithm first identifies and marks initial invalid vertices, deletes them one by one in the loop, and checks in real time whether their neighboring vertices become new invalid vertices due to a decrease in degree, adding them to the processing queue until no more vertices need to be deleted. After this process is completed, the streamlined graph data is stream-divided into independent subtasks and stored in a shared task pool. This constructs an automated, lightweight graph data optimization pipeline, which not only significantly reduces the processing load of all subsequent computing units by removing redundant data, but also avoids the overhead of global scanning through its iterative and local update mechanism, laying the data foundation for the real-time response and high scalability of the entire system.

[0036] In some embodiments, step S20 specifically includes the following steps: Each computing thread on the CPU and each computing core on the FPGA enters a ready state and concurrently accesses the shared task pool located in the CPU main memory; each computing unit performs a pull operation, acquires the access lock of the shared task pool, reads the task descriptor currently indicated by the head pointer as the subtask to be processed, then updates the head pointer to point to the next task descriptor to be processed, and releases the access lock; when the computing unit performing the pull operation is an FPGA computing core, after acquiring the task descriptor of the subtask to be processed, it locates the corresponding graph data segment from the optimized graph data based on the data range index contained in the task descriptor. Simultaneously, according to the graph topology, it determines and reads the associated data necessary to ensure the semantic integrity of the graph computing kernel function executed on the corresponding subtask, which is associated with the graph data segment; and transmits the graph data segment and associated data to the onboard memory corresponding to the FPGA computing core that initiated the pull operation via the system bus, completing the data deployment before computation.

[0037] Specifically, all computing units, as peer participants, competitively acquire tasks using mutex access locks. Units with faster processing speeds receive more tasks, thus eliminating the prediction bias of static partitioning. For FPGA computing units, after acquiring the task descriptor, they proactively locate and extract all the data required for computation based on the index in the descriptor, including associated data to ensure the semantic integrity of subtask computations, and preload it to the onboard memory via the bus. This series of steps forms a transition from "task contention" to "data readiness": it not only greatly improves the overall resource utilization and real-time response of the system through a decentralized pull model, but also effectively hides data transmission latency through a precise data preloading strategy tailored for the FPGA, preparing it for subsequent pipelined parallel computing by the FPGA. It is a key scheduling link for the efficient operation of the entire collaborative computing process.

[0038] In some embodiments, step S30 specifically includes the following steps: based on the data range index recorded in the task descriptor retrieved by the CPU computing unit, locate and load the corresponding graph data segment from the optimized graph data in main memory into the CPU cache; for the computing units to be processed in the graph data segment, obtain the corresponding set of data to be compared; according to the size characteristics of the set of data to be compared, dynamically select to execute the merge algorithm or the binary search algorithm for core computing operations, and use single instruction multiple data stream instructions to vectorize and accelerate the execution of the selected algorithm; after completing the processing of all computing units in the graph data segment, summarize the computing results and generate the first local result.

[0039] To maximize CPU computational efficiency, this embodiment employs an adaptive hybrid intersection algorithm, which considers the two neighbor sets to be compared. and The scale dynamic selection strategy. Let... And m ≥ n. The algorithm makes decisions by calculating the selection parameter α: ; when When the value is greater than 1, it indicates that the size difference between the two sets is relatively small, and the merge-based algorithm has lower comparison overhead; otherwise, the binary search-based algorithm is more efficient. This decision mechanism ensures that the core operation can achieve near-optimal performance under various data distributions.

[0040] Specifically, the system accurately locates and loads data into the cache based on the task descriptor. Then, based on the size and characteristics of the dataset to be processed, it intelligently and dynamically selects between a merge sort algorithm and a binary search algorithm to adapt to different data distributions. Furthermore, it utilizes the Single Instruction Multiple Data (SIMD) instruction set to vectorize and accelerate the execution of the selected algorithm, ultimately summarizing and generating local results. This fully leverages the CPU's advantages in complex logical judgments and vectorized computation. The adaptive algorithm selection mechanism ensures efficient computation under various data characteristics, and the combination of hardware-level parallel instructions greatly improves the throughput of core computational operations, significantly accelerating the CPU's processing speed for subtasks.

[0041] Furthermore, step S30 specifically includes the following steps: Based on the FPGA computing unit, the graph data segment and associated data are prefetched from the corresponding onboard memory to the on-chip cache of the FPGA in a burst transmission manner; based on the data in the cache, multiple parallel binary search pipeline processing units are started, and each processing unit performs the following operations according to the pipeline cycle: read the query data to be processed from the cache; perform binary search in the target data set read from the on-chip memory to perform set intersection calculation or pattern matching calculation to obtain the calculation result; accumulate the calculation result into the local accumulator corresponding to the corresponding processing unit; after all parallel pipeline processing units have completed the calculation of the allocated data, the results of the local accumulators of each processing unit are reduced to generate a second local result.

[0042] Specifically, computational data is preloaded into the on-chip cache via burst transfers to reduce access latency. Multiple parallel binary search pipeline processing units are deployed, each performing data reading, set search, and result accumulation operations in a pipelined manner. Finally, the results of all parallel units are reduced. This deeply unlocks the hardware potential of the FPGA: through a customized binary search pipeline design, computational tasks are transformed into highly regular and continuous data flow operations, fully leveraging the advantages of FPGA in fine-grained parallelism and pipeline throughput, greatly improving the execution efficiency and energy efficiency of core graph computation operations.

[0043] In some embodiments, step S40 specifically includes the following steps: based on the second local results generated by each FPGA computing unit, the respective second local results are directly written into a pre-allocated, unified result buffer in the CPU main memory via the system bus; after the result feedback is completed or a preset synchronization point is reached, a main control thread of the CPU is responsible for collecting the second local results fed back by all FPGA computing units from the result buffer, and collecting the first local results generated by each CPU computing thread; according to the main control thread of the CPU, a global aggregation operation is performed on the collected first local results and second local results, the global aggregation operation includes at least one of accumulation, merging or reduction operations, to generate the final calculation result of the large-scale graph data.

[0044] Specifically, the computation results from each FPGA unit are efficiently aggregated to a unified buffer in the CPU's main memory via the system bus. Then, the CPU's main control thread synchronously collects the local results generated by all computation units, and finally performs a global aggregation operation to generate the final result. This provides a reliable result integration mechanism for distributed heterogeneous computing systems. Through centralized result buffers and coordination with the main control thread, it ensures that computation results distributed across the CPU and FPGA are completely and orderly aggregated, and the final aggregation operation restores the correct result with global semantics, thus constituting a process from parallel computation to unified output.

[0045] The implementation principle of a real-time graph data computing method based on big data in this application embodiment is as follows: The system converts the original graph data into a compressed sparse row format and performs iterative filtering to remove redundancy, thereby obtaining optimized graph data. Then, using an edge-based partitioning strategy, the computing task is stream-divided into multiple independent subtasks, each stored as a lightweight descriptor in a shared task pool in the CPU's main memory. Next, based on a dynamic cooperative scheduling mechanism, CPU threads and FPGA cores, as peer computing units, actively and concurrently pull tasks from the shared pool. This pull-based model naturally achieves dynamic load balancing based on the actual processing speed of each unit, realizing adaptive task allocation at runtime, greatly improving the utilization of computing resources, and further enhancing real-time processing performance. For tasks pulled by the FPGA, the system transmits its corresponding data segments and associated data necessary to ensure computational integrity to the FPGA's onboard storage via a bus. During the execution phase, the CPU unit executes an optimized kernel function using the SIMD instruction set and dynamic algorithm selection, while the FPGA unit executes a customized hardware kernel function based on deep pipeline and parallel binary search, fully leveraging the advantages of their respective architectures. Finally, all the scattered local results are fed back and aggregated to generate the final computation result. Thus, through a systematic design of dynamic scheduling, heterogeneous collaboration, and joint hardware and software optimization, the uneven load caused by static partitioning and the low energy efficiency caused by a single hardware architecture in traditional methods are effectively reduced, achieving low-latency and high-energy-efficiency real-time processing of large-scale graph data.

[0046] Figure 1 This is a flowchart illustrating a real-time computation method for large-scale graph data based on big data in one embodiment. It should be understood that, although... Figure 1 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows; unless explicitly stated otherwise, there is no strict order requirement for the execution of these steps, and they can be executed in other orders; and Figure 1 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0047] Based on the same technical concept, referring to Figure 2 This application also provides a real-time computing device for large-scale graph data based on big data, which adopts the following technical solution: The device includes: The data processing module receives raw large-scale graph data, converts it into a compressed sparse row format, and filters out redundant data to obtain optimized graph data. Based on the optimized graph data, an edge-based partitioning strategy is adopted to stream-divide the computation task into multiple independent subtasks. A task descriptor containing its data range index is generated for each subtask. All task descriptors are injected into a shared task pool located in the CPU main memory. The shared task pool is configured as a circular queue structure that supports concurrent access by multiple computing units. The task allocation module is used to use one or more computing threads on the CPU side and one or more computing cores on the FPGA side as computing units to concurrently pull task descriptors to be processed from the shared task pool. For the subtasks pulled by the FPGA computing units, the graph data segment corresponding to the subtask and the associated data necessary to ensure the integrity of the calculation are transmitted to the onboard memory of the FPGA via the bus. The task processing module is used by the CPU computing unit to access the corresponding graph data segment from main memory according to the retrieved task descriptor, execute the first graph computing kernel function optimized for the CPU architecture, and obtain the first local result; the FPGA computing unit accesses data from its onboard and on-chip memory, executes the second graph computing kernel function optimized for the FPGA pipeline architecture, and the second graph computing kernel function accelerates the calculation through deep pipeline and parallel processing to obtain the second local result. The result aggregation module is used to send the second local results generated by each FPGA computing unit back to the CPU main memory. The CPU then aggregates the first and second local results from all computing units to generate the final calculation result of the large-scale graph data.

[0048] In some embodiments, the data processing module is specifically used to receive raw large-scale graph data and convert it into a compressed sparse row format. The compressed sparse row format includes a vertex offset array and a neighbor vertex identifier array to optimize storage and adjacency query efficiency. An iterative filtering algorithm is performed on the compressed sparse line format graph data to remove redundant vertices and their associated edges that do not meet the preset calculation conditions, and to generate optimized graph data with reduced size. Based on the neighbor vertex identifier array of the optimized graph data, an edge-based partitioning strategy is adopted to divide the entire computation task into multiple subtasks. Each subtask corresponds to a continuous and non-overlapping index interval in the array. For each subtask obtained from the division, a corresponding task descriptor is generated. The task descriptor at least records the start index, end index and range of vertices involved in the corresponding subtask in the neighbor vertex identifier array. The generated task descriptors are stored sequentially into a shared task pool, which is constructed as a circular queue structure that supports concurrent access, and maintains head and tail pointers to manage task status.

[0049] In some embodiments, the data processing module is specifically used to traverse all vertices in the compressed sparse row format graph data and add vertices that do not meet the preset calculation conditions to a deletion queue. The preset calculation conditions include at least that the degree of the vertex is not less than a preset threshold. When the queue to be deleted is not empty, the iterative process is executed. Each iteration includes: taking a current vertex to be deleted from the queue; removing the current vertex to be deleted and all its associated edges from the graph data structure; updating the degree of each neighboring vertex of the current vertex to be deleted. If the degree of the neighboring vertex after the update does not meet the preset calculation conditions, the corresponding neighboring vertex is added to the queue to be deleted. Iteration Termination and Output: Repeat the iteration process until the queue to be deleted is empty, and use the filtered graph data as the optimized graph data after scaling down.

[0050] In some embodiments, the task allocation module is specifically used to enable each computing thread on the CPU side and each computing core on the FPGA side to enter a ready state and concurrently access the shared task pool located in the CPU main memory. Each computing unit performs a pull operation, acquires an access lock for the shared task pool, reads the task descriptor currently pointed to by the head pointer as a subtask to be processed, then updates the head pointer to point to the next task descriptor to be processed, and releases the access lock. When the computing unit performing the pull operation is an FPGA computing kernel, after obtaining the task descriptor of the subtask to be processed, the corresponding graph data segment is located from the optimized graph data based on the data range index contained in the task descriptor. At the same time, according to the graph topology, the associated data related to the graph data segment and the necessary associated data to ensure the semantic integrity of the graph computing kernel function executed on the corresponding subtask are determined and read. The graph data segment and associated data are transmitted via the system bus to the onboard memory corresponding to the FPGA computing core that initiated the pull, thus completing the data deployment before computation.

[0051] In some embodiments, the task processing module is specifically used to locate and load the corresponding graph data segment from the optimized graph data in main memory to the CPU cache based on the data range index recorded in the task descriptor retrieved by the CPU computing unit. For the computational units to be processed in the graph data segment, obtain the corresponding set of data to be compared; based on the size characteristics of the set of data to be compared, dynamically select to execute the merge algorithm or the binary search algorithm for the core computational operation, and use single instruction multiple data stream instructions to vectorize and accelerate the execution of the selected algorithm; After processing all computational units within the graph data segment, the computational results are summarized to generate the first partial result.

[0052] In some embodiments, the task processing module is specifically used to prefetch graph data segments and associated data from the corresponding onboard memory to the on-board cache of the FPGA chip in a burst transmission manner based on the FPGA computing unit. Based on the data in the cache, multiple parallel binary search pipeline processing units are started. Each processing unit performs the following operations according to the pipeline cycle: read the query data to be processed from the cache; perform binary search in the target data set read from on-chip memory to perform set intersection calculation or pattern matching calculation to obtain the calculation result; and accumulate the calculation result into the local accumulator corresponding to the corresponding processing unit. After all parallel pipeline processing units have completed the calculation of the allocated data, the results of the local accumulators of each processing unit are reduced to generate the second local result.

[0053] In some embodiments, the result aggregation module is specifically used to write the second local results generated by each FPGA computing unit directly into a pre-allocated, unified result buffer in the CPU main memory via the system bus. After the result feedback is completed or the preset synchronization point is reached, a main control thread of the CPU is responsible for collecting the second local results fed back by all FPGA computing units from the result buffer, and collecting the first local results generated by each CPU computing thread. The CPU's main control thread performs a global aggregation operation on the collected first and second local results. The global aggregation operation includes at least one of the following: accumulation, merging, or reduction operations, to generate the final calculation result of the large-scale graph data.

[0054] This application also discloses a control device.

[0055] Specifically, the control device includes a memory and a processor, with the memory storing computer programs that can be loaded and executed by the processor to perform the aforementioned real-time computation method for large-scale graph data based on big data.

[0056] This application also discloses a computer-readable storage medium.

[0057] Specifically, the computer-readable storage medium stores a computer program that can be loaded by a processor and executed, such as the aforementioned method for real-time computation of large-scale graph data based on big data. The computer-readable storage medium includes, for example, various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0058] The above are all preferred embodiments of this application, and are not intended to limit the scope of protection of this application. Therefore, all equivalent changes made in accordance with the structure, shape and principle of this application should be covered within the scope of protection of this application.

Claims

1. A method for real-time computation of large-scale graph data based on big data, characterized in that, include: Receive raw large-scale graph data, convert it into a compressed sparse row format and filter out redundant data to obtain optimized graph data; Based on the optimized graph data, an edge-based partitioning strategy is adopted to divide the computation task into multiple independent subtasks in a streaming manner. A task descriptor containing its data range index is generated for each subtask. All the task descriptors are injected into a shared task pool located in the CPU main memory. The shared task pool is configured as a circular queue structure that supports concurrent access by multiple computing units. One or more computing threads on the CPU and one or more computing cores on the FPGA are used as computing units to concurrently pull the task descriptors to be processed from the shared task pool. For the subtasks pulled by the FPGA computing units, the graph data segment corresponding to the subtask and the associated data necessary to ensure computing integrity are transmitted to the onboard memory of the FPGA via the bus. The CPU computing unit accesses the corresponding graph data segment from main memory based on the retrieved task descriptor, executes the first graph computing kernel function optimized for the CPU architecture, and obtains the first local result; the FPGA computing unit accesses data from its onboard and on-chip memory, executes the second graph computing kernel function optimized for the FPGA pipeline architecture, and the second graph computing kernel function accelerates the calculation through deep pipeline and parallel processing to obtain the second local result. The second local results generated by each FPGA computing unit are sent back to the CPU main memory. The CPU then aggregates the first local results and the second local results from all computing units to generate the final calculation result of the large-scale graph data.

2. The method for real-time computation of large-scale graph data based on big data according to claim 1, characterized in that, The process involves receiving raw large-scale graph data, converting it into a compressed sparse row format, and filtering out redundant data to obtain optimized graph data. Based on the optimized graph data, an edge-based partitioning strategy is used to stream-divide the computational task into multiple independent subtasks. A task descriptor containing its data range index is generated for each subtask. All task descriptors are injected into a shared task pool located in the CPU main memory. This shared task pool is configured as a circular queue structure supporting concurrent access from multiple computational units, including: Receive raw large-scale graph data and convert it into a compressed sparse row format. The compressed sparse row format includes a vertex offset array and a neighbor vertex identifier array to optimize storage and adjacency query efficiency. An iterative filtering algorithm is performed on the compressed sparse line format graph data to remove redundant vertices and their associated edges that do not meet the preset calculation conditions, and to generate optimized graph data with reduced size. Based on the neighbor vertex identifier array of the optimized graph data, an edge-based partitioning strategy is adopted to divide the entire computation task into multiple subtasks. Each subtask corresponds to a continuous and non-overlapping index interval in the array. For each subtask obtained by the division, a corresponding task descriptor is generated. The task descriptor at least records the start index, end index and range of vertices involved in the corresponding subtask in the neighbor vertex identifier array. The generated task descriptors are stored sequentially into the shared task pool, which is constructed as a circular queue structure that supports concurrent access and maintains head and tail pointers to manage task status.

3. The method for real-time computation of large-scale graph data based on big data according to claim 2, characterized in that, The iterative filtering algorithm performed on the compressed sparse row format graph data removes redundant vertices and their associated edges that do not meet the preset calculation conditions, generating optimized graph data with reduced size, including: Traverse all vertices in the compressed sparse row format graph data, and add vertices that do not meet the preset calculation conditions to a deletion queue. The preset calculation conditions include at least the degree of the vertex not being less than a preset threshold. When the queue to be deleted is not empty, an iterative process is executed. Each iteration includes: taking a current vertex to be deleted from the queue to be deleted; removing the current vertex to be deleted and all its associated edges from the graph data structure; updating the degree of each neighboring vertex of the current vertex to be deleted; if the degree of the neighboring vertex after the update does not meet the preset calculation conditions, then adding the corresponding neighboring vertex to the queue to be deleted. Iteration Termination and Output: Repeat the iteration process until the queue to be deleted is empty, and use the filtered graph data as the optimized graph data after scaling down.

4. The method for real-time computation of large-scale graph data based on big data according to claim 3, characterized in that, The process involves using one or more computing threads on the CPU and one or more computing cores on the FPGA as computing units to concurrently pull task descriptors to be processed from the shared task pool. For subtasks pulled by the FPGA computing units, the graph data segment corresponding to the subtask and its associated data necessary to ensure computational integrity are transmitted to the FPGA's onboard memory via a bus, including: This enables each computing thread on the CPU and each computing core on the FPGA to enter a ready state and concurrently access the shared task pool located in the CPU main memory. Each computing unit performs a pull operation to acquire the access lock of the shared task pool, reads the task descriptor currently indicated by the head pointer as the subtask to be processed, then updates the head pointer to point to the next task descriptor to be processed, and releases the access lock. When the computing unit performing the pull operation is an FPGA computing kernel, after obtaining the task descriptor of the subtask to be processed, the corresponding graph data segment is located from the optimized graph data based on the data range index contained in the task descriptor. At the same time, according to the graph topology, the associated data related to the graph data segment and the necessary associated data to ensure the semantic integrity of the graph computing kernel function executed on the corresponding subtask are determined and read. The graph data segment and associated data are transmitted via the system bus to the onboard memory corresponding to the FPGA computing core that initiated the pull, thus completing the data deployment before computation.

5. The method for real-time computation of large-scale graph data based on big data according to claim 1, characterized in that, The CPU computing unit accesses the corresponding graph data segment from main memory based on the retrieved task descriptor, executes a first graph computing kernel function optimized for the CPU architecture, and obtains a first local result, including: Based on the data range index recorded in the task descriptor retrieved by the CPU computing unit, the corresponding graph data segment is located and loaded into the CPU cache from the optimized graph data in main memory; For the computational units to be processed in the graph data segment, obtain the corresponding set of data to be compared; based on the size characteristics of the set of data to be compared, dynamically select to execute the merge algorithm or the binary search algorithm for the core computational operation, and use single instruction multiple data stream instructions to vectorize and accelerate the execution of the selected algorithm; After processing all computational units within the graph data segment, the computational results are summarized to generate the first local result.

6. The method for real-time computation of large-scale graph data based on big data according to claim 4, characterized in that, The FPGA computing unit accesses data from its onboard and on-chip memory, executes a second graph computation kernel function optimized for the FPGA pipeline architecture, and accelerates computation through deep pipeline and parallel processing to obtain a second local result, including: Based on the FPGA computing unit, the graph data segment and associated data are prefetched from the corresponding onboard memory to the on-chip cache of the FPGA in a burst transfer manner; Based on the data in the cache, multiple parallel binary search pipeline processing units are started. Each processing unit performs the following operations according to the pipeline cycle: read the query data to be processed from the cache; perform binary search in the target data set read from on-chip memory to perform set intersection calculation or pattern matching calculation to obtain the calculation result; and accumulate the calculation result into the local accumulator corresponding to the corresponding processing unit. After all parallel pipeline processing units have completed the calculation of the allocated data, the results of the local accumulators of each processing unit are reduced to generate the second local result.

7. The method for real-time computation of large-scale graph data based on big data according to claim 4, characterized in that, The step of sending the second local results generated by each FPGA computing unit back to the CPU main memory, and then having the CPU aggregate the first and second local results from all computing units to generate the final calculation result of the large-scale graph data, includes: Based on the second local results generated by each FPGA computing unit, the respective second local results are directly written into the pre-allocated, unified result buffer in the CPU main memory via the system bus; After the result feedback is completed or the preset synchronization point is reached, a main control thread of the CPU is responsible for collecting the second local results fed back by all FPGA computing units from the result buffer, and collecting the first local results generated by each CPU computing thread. The CPU's main control thread performs a global aggregation operation on the collected first local results and second local results. The global aggregation operation includes at least one of the following: accumulation, merging, or reduction operations, to generate the final calculation result of the large-scale graph data.

8. A real-time computing device for large-scale graph data based on big data, characterized in that, The device includes: The data processing module receives raw large-scale graph data, converts it into a compressed sparse row format, and filters out redundant data to obtain optimized graph data. Based on the optimized graph data, it adopts an edge-based partitioning strategy to stream-divide the computation task into multiple independent subtasks, generates a task descriptor containing its data range index for each subtask, and injects all the task descriptors into a shared task pool located in the CPU main memory. The shared task pool is configured as a circular queue structure that supports concurrent access by multiple computing units. The task allocation module is used to use one or more computing threads on the CPU side and one or more computing cores on the FPGA side as computing units to concurrently pull the task descriptors to be processed from the shared task pool. For the subtasks pulled by the FPGA computing units, the graph data segment corresponding to the subtask and the associated data necessary to ensure the integrity of the calculation are transmitted to the onboard memory of the FPGA via the bus. The task processing module is used by the CPU computing unit to access the corresponding graph data segment from main memory according to the retrieved task descriptor, execute the first graph computing kernel function optimized for the CPU architecture, and obtain the first local result; the FPGA computing unit accesses data from its onboard and on-chip memory, executes the second graph computing kernel function optimized for the FPGA pipeline architecture, and the second graph computing kernel function accelerates the calculation through deep pipeline and parallel processing to obtain the second local result. The result aggregation module is used to send the second local results generated by each FPGA computing unit back to the CPU main memory. The CPU then aggregates the first local results and the second local results from all computing units to generate the final calculation result of the large-scale graph data.

9. A control device, characterized in that, The device includes: A memory and a processor, wherein the memory stores a computer program that can be loaded by the processor and executed as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer program is stored that can be loaded by a processor and executed as described in any one of claims 1 to 7.

Images