Regular path query system and method based on in-memory computing

Abstract

The application provides a regular path query system and method based on in-memory computing, the system comprises a host end; an in-memory computing end containing a plurality of in-memory computing modules working in parallel; the system divides a graph according to nodes based on the degree characteristics of the nodes, wherein high-degree nodes with a degree exceeding a threshold value and their adjacency relations are allocated to the host end for storage and processing, and low-degree nodes with a degree not exceeding the threshold value and their adjacency relations are allocated to the in-memory computing end for storage and processing. The application offloads random access tasks of low-degree nodes to a PIM end, fully utilizes the high-bandwidth characteristics of the PIM, and at the same time, retains high-degree nodes on the host end, utilizes the cache locality of the CPU, and thus improves the query efficiency.

Description

Technical Field

[0001] This invention relates to the field of computer database and in-memory computing technology, specifically to a regular expression path query system and method based on in-memory computing. Background Technology

[0002] Graph databases use attribute graph models to represent and store data, where nodes represent entities and edges represent relationships between entities. Nodes can be described by labels, and edges by attributes. In graph databases, regular expression path queries are a core query operation designed to find pairs of nodes connected by paths that conform to specific regular expressions.

[0003] With the explosive growth of data volume, traditional CPU-based graph databases face significant performance bottlenecks when handling large-scale regular expression path queries. Because graph traversal involves numerous pointer tracing operations, its memory access patterns exhibit high irregularity and randomness, resulting in extremely low cache hit rates and requiring frequent memory accesses. This leads to frequent data movement between memory and cache, severely restricting overall system performance due to the "memory wall."

[0004] To address the memory wall problem, Processing-in-Memory (PIM) technology emerged. PIM technology embeds the processor directly into memory modules (such as DRAM), enabling computation to be performed near the data storage location. This architecture significantly accelerates data-intensive applications by reducing data movement between the host CPU and memory. Current commercial PIM systems (such as UPMEM) typically consist of a host and multiple PIM modules, each with its own on-chip processor and local memory.

[0005] Running a graph database on a PIM architecture first presents the challenge of distributing graph data across different PIM modules, i.e., graph partitioning. Common graph partitioning methods include: Hash partitioning: Nodes are randomly assigned to computing nodes using a hash function. This method is computationally simple and can guarantee balanced storage load, but it violates the topological locality of the graph, resulting in a large amount of cross-PIM module communication during queries, which is not only expensive in terms of bandwidth but also increases latency.

[0006] Greedy partitioning (such as the LDG method): This type of method attempts to assign a node to the partition containing its majority of neighbors in order to maintain locality. However, these algorithms typically require traversing all neighbors and calculating complex score functions, resulting in excessively long partitioning times and difficulty adapting to dynamic graph scenarios with frequent node insertions and deletions.

[0007] Furthermore, real-world graphs typically exhibit a power-law distribution, meaning there are a small number of highly connected "high-degree nodes" and a large number of "low-degree nodes." The on-chip processors of PIM modules generally have limited computing power and small caches. Simply assigning high-degree nodes to PIM modules would lead to severe load imbalance, where the module handling high-degree nodes becomes the bottleneck, while other modules remain idle.

[0008] Therefore, how to make good use of PIM's high parallel bandwidth and design a graph database system that can maintain graph locality, achieve load balancing, and support efficient dynamic updates is an urgent problem to be solved in the current technology field. Summary of the Invention

[0009] In view of the shortcomings of the prior art, the purpose of this invention is to provide a regular path query system and method based on in-memory computation.

[0010] According to the present invention, a regular expression path query system based on in-memory computation includes: Host side; The in-memory computing end contains multiple in-memory computing modules that work in parallel. The system divides the graph into non-overlapping nodes based on the degree characteristics of the graph nodes. Nodes with a degree exceeding a threshold and their adjacency relationships are allocated to the host for storage and processing, while nodes with a degree below the threshold and their adjacency relationships are allocated to the in-memory computing terminal for storage and processing.

[0011] Preferably, the system further includes a graph partitioning module for executing a dynamic graph partitioning strategy: When a new node is inserted, the graph partitioning module obtains the neighbor node information of the new node in the current batch or in the historical cache; Identify the in-memory computing module containing the neighbor with the highest degree among the neighbor nodes; A multi-neighbor adaptive strategy is adopted to assign the new node to the in-memory computing module where the neighbor with the highest degree is located.

[0012] Preferably, the graph partitioning module is further configured to perform load balancing: When allocating nodes, monitor the load of each in-memory computing module in real time; Set dynamic capacity constraints. When the load of the target in-memory computing module exceeds the constraint value calculated based on the average load of the entire system, prohibit the allocation of new nodes to the module, and reselect an in-memory computing module that is not overloaded and contains the neighbor of the new node for allocation.

[0013] Preferably, the system further includes a node migration module: During the graph data update process, nodes are migrated between the host and in-memory computing ends based on changes in node degree. During the path query process, monitor the next-hop hit rate of the nodes stored in the in-memory computing module, that is, the proportion of the next-hop neighbors located in the same in-memory computing module in the path query initiated by the node. For nodes whose next-hop hit rate is lower than a preset ratio, migrate them to the in-memory computing module that contains the most neighbors of that node.

[0014] Preferably, the host is configured to convert graph operations into a sequence of matrix operation-based operators and distribute them: Query transformation: The query module transforms regular path queries into a sequence of query operators that include matrix multiplication and reduction operators; Update Transformation: The graph update module transforms graph data changes into a sequence of update operators that include addition and subtraction operators; Operator execution: The in-memory computing module is equipped with an operator execution module, which is used to parse and execute the received query operator sequence and update operator sequence; Parallel filtering: The in-memory computing module is equipped with a filtering module, which is used to perform in-situ filtering on the path based on node labels or edge attributes in parallel during the stage of generating intermediate results by executing matrix multiplication operators, and only retain intermediate results that meet the filtering conditions to enter the next hop query.

[0015] Preferably, the system employs a heterogeneous graph storage mechanism: The host uses an adjacency matrix to store the next-hop neighbor list of the height node and allocates contiguous memory space for each height node. The in-memory computing terminal maintains an element position mapping table and a free position table for height number nodes in the in-memory computing module; When performing edge insertion or deletion operations on height nodes, the in-memory computing end uses the element position mapping table to query the specific position of the edge or uses the free position linked list to manage the available space, and returns the calculated position index to the host end. The host end only performs the write operation based on the position index.

[0016] A regular expression path query method based on in-memory computation according to the present invention includes the following steps: The storage process involves dividing the graph data into a set of nodes with high degree and a set of nodes with low degree, based on the degree of the graph nodes. The set of nodes with high degree is stored on the host side, while the set of nodes with low degree is distributed and stored in multiple in-memory computing modules on the in-memory computing side. Query distribution steps: The host receives a regular expression path query request and converts it into a matrix operation task; the operation tasks involving low-degree nodes are distributed to the corresponding in-memory computing modules for parallel execution, while the operation tasks involving high-degree nodes are executed on the host. Query execution steps: When performing path search, the in-memory computing module directly uses the locally stored attribute data to filter the generated intermediate results in situ according to the query rules; if the query does not reach the target number of hops and the next hop node is located in another in-memory computing module or the host, the intermediate results are exchanged; when the target number of hops is reached, the partial query results are returned to the host. Result reduction steps: After the host collects all local query results, it distributes these data to the in-memory computing module based on hash consistency to perform reduction in parallel, deduplicatizes and merges the results in parallel, and finally returns the normalized results to the host.

[0017] Preferably, the result reduction step includes: The host distributes each query result tuple to the corresponding in-memory computing module according to the hash consistency distribution rule; Each in-memory computation module maintains a hash set locally for deduplication. For each received tuple, it attempts to insert it into the hash set. If the insertion is successful, it is retained; if the insertion fails, it is discarded. After all in-memory computation modules have completed deduplication, they return all tuples in their respective hash sets to the host for merging, resulting in the final query result without duplicates.

[0018] Preferably, the method further includes a dynamic graph partitioning step: When inserting a new node, obtain the neighbor node information of the new node, and select the in-memory computing module containing the neighbor with the highest degree among the neighbors as the target module; Check if the current load of the target module exceeds the dynamic capacity constraint. If it does not exceed the constraint, assign the new node to the target module. If it does exceed the constraint, select the next in-memory computing module where the next neighbor is located in descending order of degree. If no neighboring module satisfies the constraints, the target module is selected from the set of in-memory computing modules whose load does not exceed the dynamic capacity constraints using a hash function.

[0019] Preferably, the method further includes a heterogeneous graph update step: When inserting an edge into a height node, the host distributes the edge's metadata to the corresponding in-memory computation module according to hash consistency. The in-memory computation module queries the element position mapping table to determine if the edge exists. If it does not exist, it retrieves the free position index from the free position table, updates the element position mapping table, and returns the position index to the host. If it exists, the update is terminated. The host writes the target node into a contiguous array of the adjacency matrix according to the position index. When deleting an edge from a height node, the host distributes the edge's metadata to the corresponding in-memory computation module according to hash consistency. The in-memory computation module queries the element position mapping table to obtain the position index, pushes the position index into the free position table, deletes the entry in the element position mapping table, and returns the position index to the host. The host then performs logical deletion in the contiguous array of the adjacency matrix based on the position index.

[0020] Compared with the prior art, the present invention has the following beneficial effects: 1. This invention improves query efficiency by offloading the random access task of low-degree nodes to the PIM end, making full use of the high bandwidth characteristics of PIM; at the same time, it retains high-degree nodes on the host end, taking advantage of the CPU's cache locality.

[0021] 2. This invention effectively solves the problem of unbalanced load. By using a division of labor strategy based on degree thresholds, it avoids the PIM module from processing nodes with high height, thus significantly improving the load balance of the PIM system when processing skew map data.

[0022] 3. This invention reduces communication overhead by effectively maintaining the locality of graph data through a multi-neighbor adaptive partitioning algorithm, thereby reducing cross-PIM module communication. Simultaneously, the in-memory filtering mechanism eliminates a large number of invalid paths during the query process, significantly reducing cross-PIM module communication and host-PIM communication.

[0023] 4. This invention improves graph update throughput. The heterogeneous graph storage design offloads the complex edge existence verification and storage space management tasks involved in the host-side height node update process to the PIM side for parallel processing, so that the host side only needs to perform simple memory write operations, which greatly improves the update performance of dynamic graphs. Attached Figure Description

[0024] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 This is a schematic diagram of the overall architecture of the regular expression path query system based on in-memory computation provided in this embodiment of the invention, which shows the functional module division, data interaction path and hardware-software collaborative division architecture between the host end and the PIM end. Figure 2 This is a flowchart of dynamic graph partitioning and node allocation provided in the embodiments of the present invention. It shows in detail the judgment logic and execution steps of determining the storage location (host end or PIM end) based on the degree threshold when inserting a new node, and determining the target PIM module by combining the multi-neighbor adaptive strategy and dynamic load balancing constraints when allocating to the PIM end. Figure 3This diagram illustrates the graph partitioning method and storage distribution example provided in this invention. It visually demonstrates the logic of storing high-degree nodes on the host side and low-degree nodes on the PIM side based on a node degree threshold, as well as the principle of multi-neighbor adaptive partitioning. The node label represents the subnet segment, and the edge attribute represents the channel security status, clearly illustrating the heterogeneous storage logic based on the degree threshold. Figure 3 This will serve as a reference topology for subsequent embodiments; Figure 4 This is a flowchart of the regular expression path query execution provided in an embodiment of the present invention, based on Figure 3 The topology and partitioning demonstrate the computational process of transforming a typical 2-hop path query without filtering conditions into a sequence of sparse matrix multiplication operators. Figure 5 This is a schematic diagram illustrating the node tag filtering principle provided in an embodiment of the present invention, showing that... Figure 4 Based on the operator sequence, the logic of truncating nodes of a specific "subnet segment" is achieved through a node label filtering matrix; Figure 6 This is a schematic diagram illustrating the edge attribute filtering principle provided in an embodiment of the present invention, showing that... Figure 4 Based on the operator sequence, the logic of filtering the "secure channel" is performed using the edge attribute filtering matrix; Figure 7 This is a schematic diagram of the heterogeneous graph storage mechanism provided in an embodiment of the present invention, based on... Figure 3 The topology and partitioning show the correspondence and interaction between the host-side height number node adjacency matrix and the PIM-side element position mapping table and free position table; Figure 8 This is a schematic diagram of the unloading update process for inserting height-number nodes in an embodiment of the present invention, illustrating the interaction steps and division of responsibilities between the host and PIM terminals during the edge insertion process; Figure 9 This is a schematic diagram of the unloading update process for edge deletion of height-number nodes provided in an embodiment of the present invention, which shows the interaction steps and division of responsibilities between the host and PIM terminals during the edge deletion process. Detailed Implementation

[0025] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the present invention. These all fall within the protection scope of the present invention.

[0026] I. System Overall Architecture This invention deals with attribute graphs, where nodes represent entity units and edges represent relationships between entities. Nodes can carry labels (referred to as node labels), and edges can carry attribute key-value pairs (referred to as edge attributes). Figure 1 As shown, the present invention provides a regular expression path query system based on in-memory computing. The system is based on an in-memory computing architecture and includes a host end and a PIM end.

[0027] 1. Host side The host computer is equipped with a high-performance central processing unit (CPU) and main memory. Logically, the host computer mainly includes the following functional modules: The query module is responsible for receiving users' regular expression path query requests, converting them into an execution plan based on matrix operations, and distributing the operators to the host (for processing tasks corresponding to height nodes) or the corresponding PIM module (for processing tasks corresponding to low-degree nodes). It generates matrix multiplication and reduction operators, and processes queries for height nodes.

[0028] The graph update module is responsible for receiving user graph update requests, converting them into an execution plan based on matrix operations, and distributing operators to the host (for processing tasks related to height nodes) or the corresponding PIM module (for processing tasks related to low-degree nodes). It generates addition and subtraction operators and processes the updates of height nodes simultaneously.

[0029] The graph partitioning module is responsible for determining the location of nodes during graph construction or node insertion. It maintains graph partitioning information and PIM load balancing information, and generates "node partition vectors" to quickly locate the storage location of nodes.

[0030] Node migration module: Responsible for dynamically adjusting the graph layout. It identifies and migrates newly generated high-degree nodes to the host side, low-degree nodes to the PIM side, or migrates nodes with low next-hop hit rates between PIM modules to optimize the locality of local subgraphs in PIM.

[0031] Heterogeneous storage module (host-side portion): Stores height nodes and their adjacency lists. Since height nodes exhibit good access locality, storing them on the host side can fully utilize the CPU's cache advantages.

[0032] 2. Internal Calculation Terminal (PIM Terminal) The PIM terminal consists of N parallel-operating PIM modules (e.g., PIM 0 to PIM N-1). Each PIM module is equipped with an on-chip processor and local memory. Logically, each PIM module contains: Operator execution module: Parses and executes operator tasks from the host. For query tasks, this module performs sparse matrix multiplication and deduplication; for update tasks, this module updates locally stored data.

[0033] Filtering module: When intermediate results are generated during path query, the path is filtered directly within the PIM module based on node labels or edge attributes according to regular expression conditions, and only intermediate results that meet the conditions are retained, thereby greatly reducing cross-PIM module communication and host-PIM communication.

[0034] Local storage module: Used to store low-degree nodes. This module uses a hash mapping structure to store the adjacency matrix, leveraging the PIM architecture's ability to handle nodes with high random access characteristics without requiring data movement and its high parallelism.

[0035] Heterogeneous storage auxiliary module: Stores auxiliary information about height nodes (element position mapping table, free position table) to assist the host in performing efficient height node updates.

[0036] In this invention, the (row,col) tuple is used to represent two associated node identifiers. In the context of graph storage and updating, (row,col) represents an edge in the graph, where row is the starting node and col is the ending node; in the execution of regular expression path queries, (row,col) represents the start and end node pair of the query result. Unless otherwise specified, (row,col) as used below refers to the node pair in both of the above situations.

[0037] During the system operation of this invention, to ensure the deterministic allocation of (row,col) tuples across multiple PIM modules, a hash consistency distribution rule is adopted: the host calculates a hash value for each (row,col) tuple and determines the target module number by taking the modulo of the total number of PIM modules N, i.e., PIM_ID = hash(row,col) mod N. Identical tuples are always distributed to the same PIM module, thereby ensuring the consistency of query reduction, height node insertion, and deletion operations, while also achieving load balancing among PIM modules.

[0038] II. Division of Labor Strategy for Software and Hardware Collaboration This invention employs a task allocation strategy based on a node degree threshold (Th), which assigns high-degree nodes to the host and low-degree nodes to the PIM.

[0039] Low-degree node allocation: When a node's degree is less than the threshold Th (e.g., 16, which can be dynamically configured by the system administrator based on the actual workload), the node is identified as a low-degree node and allocated to the local graph storage module on the PIM side. This type of node accounts for a high proportion of the total number of nodes, but the size of each node's neighbors is relatively small, and the data access pattern exhibits strong random characteristics, making it more suitable for the parallel processing characteristics of the PIM architecture.

[0040] Height-based node allocation: When a node's degree is greater than or equal to the threshold Th, the node is identified as a height-based node and allocated to the host. These nodes are frequently accessed and have contiguous neighbors, making them suitable for CPU caching.

[0041] Dynamic node allocation adjustment: During system operation, as nodes are inserted and deleted, the node migration module migrates newly generated high-degree nodes from the PIM end to the host end, and assigns low-degree nodes from the host end to the PIM end to maintain load balance. If an increase or decrease in the average density of the graph (the average number of neighbors per node) is detected, the threshold Th can be adaptively adjusted to adapt to the evolution of the graph structure.

[0042] III. Dynamic Graph Partitioning Method Based on Multi-Neighbor Adaptation To address the problems of existing hash partitioning methods violating graph locality and the excessive computational cost of greedy algorithms (such as LDG), this invention designs a PIM-friendly multi-neighbor adaptive dynamic graph partitioning method that can maintain the locality of the partitioned subgraphs at a lower cost. The specific process of this method is as follows: Figure 2 As shown, the specific steps are as follows: 1. Neighbor information acquisition and classification When a new node (or edge) enters the system, the graph partitioning module first checks the degree of the node. If the degree is greater than the threshold Th, it is directly assigned to the host. If the degree is less than the threshold Th, the PIM allocation process begins. For nodes assigned to PIMs, the system adopts a multi-neighbor adaptive allocation strategy, which further introduces a dynamic capacity constraint C (defined in "3. Dynamic Load Balancing Constraint") when selecting the PIM module containing the target neighbor, in order to avoid overloading a single PIM module while maintaining locality.

[0043] 2. Multi-neighbor adaptive allocation strategy Based on the neighbor information of the current batch or historical cache, the system executes the following allocation logic: Step S21: Neighbor retrieval. Obtain all known neighbor nodes of the new node, forming a neighbor set N (temporary variable).

[0044] Step S22: Select the neighbor with the highest degree. In the known neighbor set N, select the neighbor node with the highest degree (called the target neighbor) in descending order of neighbor degree.

[0045] Step S23: Follow-up assignment. Query the PIM module ID where the target neighbor is currently located, and directly assign the new node to that PIM module.

[0046] Step S24: Hash rollback. If the new node has no known neighbors (e.g., the first node in the batch), the rollback uses a hash function to randomly assign a PIM module.

[0047] 3. Dynamic load balancing constraints To prevent certain PIM modules from becoming overloaded due to the "follow-allocation" strategy, the system introduces a dynamic capacity constraint mechanism.

[0048] The system maintains the node load L of each PIM module in real time. i Set the dynamic constraint threshold C = α × L avg L avg The average load of all PIM modules is calculated as the total number of allocated nodes divided by the number of PIM modules. α is the load balancing slack factor, used to control the system's tolerance for load imbalance, thereby achieving a dynamic balance between data locality and storage balance. When α=1, it indicates that the system strictly enforces load balancing, and the load of any module cannot exceed the average value. When α>1, it indicates that the system allows a certain degree of load skew to prioritize maintaining the locality of graph data. The larger the α value, the higher the tolerance for load imbalance, and the greater the space for locality optimization. In this embodiment, the initial value of α is set to 1.1, meaning that under the premise of prioritizing locality, the module load is allowed to not exceed 10% of the average value. This parameter can be dynamically adjusted by the system administrator according to the actual workload characteristics (e.g., adjusted to 1.3 in a locality-first scenario and to 1.05 in a balance-first scenario).

[0049] Before performing step S23, check the load L of the target PIM module. target Does it exceed C? If L target If the value is >C, then the target module is abandoned, the neighbor is removed from the set N, and the next candidate neighbor's PIM module is selected in descending order of degree.

[0050] If no neighbor of the target PIM is found that does not exceed the load, proceed to step S24, and select the target module from the set of PIM modules whose load does not exceed the dynamic constraint threshold C by using a hash function.

[0051] After each successful node allocation, the system recalculates the average load L based on the updated load of each PIM module. avg And update the dynamic constraint threshold C accordingly.

[0052] In one embodiment of the present invention, such as Figure 3 As shown, this diagram illustrates the partitioning of a directed network topology in a system with one host and two PIM modules. The network topology is input sequentially from node 0 to node 9, with α set to 1.4 and the node threshold Th set to 2. The partitioning process is as follows: Node 0 is assigned to PIM 0 using hash rollback, and L0 is updated to 1. At this time, the dynamic constraint threshold C is 0.7. Nodes 1 and 2 are height nodes, assigned to the host side; When node 3 attempts to assign to PIM 0 based on the neighbor with the highest degree, because PIM 0 exceeds the dynamic constraint threshold (L0>C), it uses hash backoff in the remaining PIM modules and assigns it to PIM 1, updating L1=1. At this time, the dynamic constraint threshold C=1.4. Node 4 adopts a multi-neighbor adaptive allocation strategy. The neighbor with the highest degree in PIM is node 0, so it is assigned to PIM0 to which node 0 belongs, and L0 is updated to 2. At this time, the dynamic constraint threshold C is 2.1. Node 5 adopts a multi-neighbor adaptive allocation strategy. The neighbor with the highest degree in PIM is node 0, so it is assigned to PIM0 to which node 0 belongs, and L0 is updated to 3. At this time, the dynamic constraint threshold C is 2.8. When node 6 attempts to assign to PIM0 based on the neighbor with the highest degree, because PIM0 exceeds the dynamic constraint threshold (L0>C), it uses hash backoff in the remaining PIM modules and assigns it to PIM1, updating L1=2. At this time, the dynamic constraint threshold C=3.5. Node 7 adopts a multi-neighbor adaptive allocation strategy. The neighbor with the highest degree in PIM is node 3, so it is assigned to PIM1 to which node 3 belongs, and L1 is updated to 3. At this time, the dynamic constraint threshold C is 4.2. Node 8 adopts a multi-neighbor adaptive allocation strategy. The neighbor with the highest degree in PIM is Node 5, so it is assigned to PIM0 to which Node 5 belongs, and L0 is updated to 4. At this time, the dynamic constraint threshold C is 4.9. Node 9 adopts a multi-neighbor adaptive allocation strategy. The neighbor with the highest degree in PIM is Node 6, so it is assigned to PIM1 to which Node 6 belongs, and L1 is updated to 4. At this time, the dynamic constraint threshold C is 5.6.

[0053] The node attribute in the graph is "subnet segment", and the edge attribute is "whether it is a secure channel". The entire graph is stored in the form of an adjacency list, and the adjacency list of all nodes can be abstracted into an adjacency matrix.

[0054] 4. Dynamic node migration The graph partitioning method provided by this invention is not a static, one-time process. During system operation (including query execution and graph update phases), the graph partitioning module continuously monitors node degree changes and access locality statistics to identify nodes with unreasonable partitioning. For these nodes, the system will perform dynamic migration operations to correct the initial partitioning or adapt to the evolution of the graph structure. The specific triggering conditions, judgment logic, and execution flow for node migration will be described in detail in Section VI, "Dynamic Node Migration Strategy," below.

[0055] IV. Regular Expression Path Query Process This invention abstracts regular path query into a series of matrix-based operations and utilizes the collaborative work of the host and PIM ends to achieve efficient parallel processing.

[0056] 1. Query plan generation and matrix operation logic When the query module on the host receives a regular expression path query request, it first parses the query statement and converts the query path pattern into a sequence of operators based on matrix operations.

[0057] Unfiltered queries are transformed into a sequence of sparse matrix multiplication operators. In one embodiment of the invention, a 2-hop unfiltered query can be represented by the mathematical formula Result = Q × Adj × Adj, as follows: Figure 3 As shown in the diagram. Here, Q is a vector or matrix representing the source node (using one-hot encoding), and Adj is the adjacency matrix of the graph. The PIM module performs a hop-by-hop expansion search of the path by continuously performing matrix multiplication.

[0058] Filtered Query: The query module embeds specific algebraic operation instructions into the operator sequence. Node Label Filtering: Defined as a matrix multiplication operation (Adj×NF). Here, NF is the node label filtering matrix, used to filter nodes before path expansion. Edge Attribute Filtering: Defined as a logical AND operation (Adj&EF). Here, EF is the edge attribute filtering matrix, used to remove edges that do not meet the attribute conditions.

[0059] Multi-hop reduction: For multi-hop queries, the query module appends a reduction operator to the end of the operator sequence to aggregate and deduplicate the final results, preventing duplicate statistics. Furthermore, for cases with a particularly large number of hops (e.g., more than 4 hops), to control the size of the intermediate result set during the query process, the reduction operator is not limited to execution at the end of the query; it can also be configured to trigger the reduction operation in the middle stages of multi-step path iterations, based on a preset threshold for the size of the intermediate result set, thereby effectively suppressing the explosive growth of the number of paths.

[0060] 2. Task distribution and parallel execution The query module decomposes and distributes the query task to the host and each PIM module based on the "node partitioning vector" stored by the graph partitioning module. Specifically, the query matrix Q is split into host terminal submatrices Q. host and the submatrix Q corresponding to each PIM module pim_i This ensures that each computing unit only processes the matrix rows associated with its local storage node.

[0061] Host-side execution: Responsible for calculating the matrix multiplication part involving height nodes (processing Q). host Because the high-order nodes are stored contiguously on the host side, the CPU can efficiently perform sequential memory access, leveraging the advantages of caching to accelerate computation.

[0062] PIM execution: Responsible for calculating the matrix multiplication part involving low-degree nodes (processing Q). pim_i Each PIM module utilizes its internal operator execution module to concurrently process the local subgraph on the memory side, eliminating the bandwidth bottleneck caused by large-scale access to low-degree nodes.

[0063] 3. In-memory query and in-situ filtering When performing matrix multiplication on the query execution side (mainly the PIM module) to find the next-hop node, the operator execution module will perform the corresponding operation according to the type of operator received.

[0064] The response matrix multiplication operator receives the query matrix, where non-zero elements represent the source node `row`. It then indexes the corresponding row in the adjacency matrix based on `row`'s identifier, retrieves all its next-hop nodes `col`, and returns (row, col). Logically, this is equivalent to performing Q×Adj. The system achieves hop-by-hop topology expansion of the regularized path in an in-memory environment by using the set of `col` output from the current iteration as the input `row'` for the next iteration.

[0065] Response node label operator: When accessing the adjacency matrix Adj to obtain the next hop, check the node label data. This is logically equivalent to performing Adj × NF; that is, if the node label does not match, set its state in the path to zero and directly truncate the path. In one embodiment of the present invention, as... Figure 5 As shown, the regular path condition is "the node belongs to subnet 1". A 2-hop regular query with this condition can be represented by the mathematical formula Result=Q×(Adj×NF)×(Adj×NF).

[0066] Responding to edge attribute operators: When traversing the adjacency matrix Adj to obtain the next hop, the edge attribute data is checked. This is logically equivalent to performing Adj&EF, that is, the path is allowed only if the edge attribute meets the condition; otherwise, the edge is considered non-existent (the logical AND result is 0). In one embodiment of the invention, as... Figure 6 As shown, the regular expression path condition is "through a secure channel". A 2-hop regular expression query with this condition can be represented by the mathematical formula Result=Q×(Adj&EF)×(Adj&EF).

[0067] Multi-hop intermediate result processing: If the target node for the next hop is located in another PIM module or on the host before the target query hop count is reached, the current intermediate result is written to the intermediate result buffer and sent to the target location. If the next hop node is still located in the current PIM module, the query continues to iterate locally, and the result is written to the local result buffer after the query is completed. The local result of each query consists of several (row, col) tuples, where row represents the starting node ID of the path and col represents the ending node ID of the path.

[0068] Technical effect: By performing in-situ filtering on the query execution side, as the number of query hops increases, a large number of paths that do not meet the conditions are truncated at the data source, exponentially reducing the number of intermediate results that need to be transmitted across modules, thereby significantly reducing the overhead of cross-PIM module communication and host-PIM communication.

[0069] 4. Result Reduction After each query execution side completes its query, the host side collects all local results and initiates the parallel reduction process: Step S41: Hash Distribution. The host side, according to the aforementioned hash consistency distribution rule, traverses all local query results and assigns each (row,col) tuple to the corresponding PIM module.

[0070] Step S42: Local deduplication at the PIM end. After each PIM module receives a batch of (row, col) tuples, it maintains a temporary hash set for deduplication locally. For each received tuple, the PIM module performs: Try inserting (row, col) into this hash set; If the insertion is successful (i.e., the tuple has not appeared before), it is retained; If insertion fails (i.e., the tuple already exists in the set), it is discarded without any further processing.

[0071] In this way, each PIM module independently completes efficient deduplication of a portion of the data, and because step S41 ensures that the same tuple will only appear in the same module, it ensures that the same (row,col) pair is only retained once.

[0072] Step S43: Result Summarization. After all PIM modules have completed deduplication, they return all tuples in their respective hash sets to the host. The host directly merges these tuples to obtain the final unique query result, which is then returned to the user as the response to the regular expression path query.

[0073] Technical effects: By offloading the deduplication operation to the PIM end for parallel execution, the host end is avoided from becoming a bottleneck for data aggregation; by using key-based hash distribution, the problem of repeated processing of the same result across modules is fundamentally avoided; by utilizing the O(1) insertion and deduplication characteristics of hash sets, high-performance parallel reduction is achieved, which significantly improves the processing efficiency of large-scale result sets.

[0074] V. Heterogeneous Graph Storage and Update Mechanism To address the complex space management overhead faced by the host side when handling height node updates in dynamic graph scenarios, this invention designs a heterogeneous graph storage mechanism. This mechanism retains data storage on the host side, while offloading metadata management and update logic to the PIM (Graph Information Modeling) side. In one embodiment of this invention, the host stores two height nodes, and their heterogeneous storage is as follows: Figure 7 As shown.

[0075] 1. Storage Structure Design On the host side: Maintain the adjacency matrix of the height-counting nodes. For each height-counting node, allocate a contiguous array cols_vector to store the IDs of its next-hop neighbors. This structure ensures efficient sequential reading during queries.

[0076] PIM Side: Maintains two auxiliary hash mapping tables to assist in update management on the host side: The element position mapping table stores the mapping relationship from "edge" to "host-side array index", i.e., <edge, position>. The free position table stores the list of array indices in the host-side cols_vector that are currently empty (e.g., holes left after element deletion).

[0077] 2. Uninstallation-based image update process For edge insertion operations on height-number nodes, the update process is as follows: Figure 8 As shown below, the unloading update method of the present invention is illustrated using the insertion of edge (u,v) as an example (where u is the height node).

[0078] Step S51: Hash Distribution. The host distributes the metadata request for edge (u,v) to the corresponding PIM module PIM k according to the aforementioned hash consistency distribution rule.

[0079] Step S52: Existence check. PIM k queries the local element location mapping table to determine if an entry <(u,v),pos> already exists. If it exists, a preset flag value (e.g., -1) is returned, notifying the host to terminate this update.

[0080] Step S53: Space Allocation. If the edge does not exist, PIM k locates the free management entry corresponding to node u in the free location table; if no match is found, a new free management entry is allocated for node u and initialized. Subsequently, PIM k retrieves an element from the available location list maintained by this entry as the free location index pos; pos can be a reused hole location index, or a tail location index appended by this entry when the hole set is empty.

[0081] Step S54: Metadata update. PIM k records the new entry <(u,v),pos> in the element location mapping table, and then returns the location index pos to the host.

[0082] Step S55: Locate the adjacency matrix row. The host searches for the adjacency matrix row (i.e., cols_vector) of node u in the adjacency matrix. If not found, a new cols_vector is allocated for node u and initialized.

[0083] Step S56: Host write. The host checks the value of pos. If it is the preset flag value, the update is terminated. If pos exceeds the allocated capacity of the current adjacency matrix row, cols_vector is expanded, and finally the target node v is written to cols_vector[pos].

[0084] In one specific embodiment of the present invention, reference is made to... Figure 7 As shown, the mapping table on the PIM side adopts a hash bucket structure with slotted storage. This design allows the system to quickly locate the free management table entry corresponding to node u through the hash function when inserting a new edge, and generate pos with O(1) hole pop or tail increment, thereby avoiding linear scanning and frequent relocation of the adjacency array on the host side, reducing update overhead and improving parallelism.

[0085] In one specific embodiment of the present invention, combined with Figure 7 and Figure 8 Taking the insertion of an edge (1,2) pointing to node 2 into node 1 as an example, the specific update interaction process is as follows: The host distributes the edge via hash and passes it to PIM 0 for processing. After confirming that the edge does not exist in the element position mapping table, PIM 0 pops the free position index pos=1 (corresponding to the -1 placeholder in the host array) from the free position table entry corresponding to node 1. Then it updates the local element position mapping table and writes a new key-value pair <(1,2),1>. PIM 0 then returns the position index pos=1 to the host, and the host directly writes 2 into cols_vector[1] to complete the insertion.

[0086] For edge deletion operations on height-number nodes, the process is symmetrical to that of insertion operations in terms of position management. The update process is as follows: Figure 9 As shown below, the following explanation uses the deletion of edge (u,v) as an example.

[0087] Step S61: Hash Distribution. The host distributes the metadata request for edge (u,v) to the corresponding PIM module PIM k according to the aforementioned hash consistency distribution rule.

[0088] Step S62: Existence check. PIM k queries the local element location mapping table to determine if an entry <(u,v),pos> already exists. If it does not exist, a preset flag value (e.g., -1) is returned, notifying the host to terminate this update.

[0089] Step S63: Location reclamation and metadata removal. If the edge exists, obtain the location index pos corresponding to the entry and push pos into the idle management table entry corresponding to node u to reclaim the location index. PIM k deletes the entry <(u,v),pos> from the element location mapping table and then returns the location index pos to the host.

[0090] Step S64: Locate the adjacency matrix row. The host searches for the adjacency matrix row (i.e., cols_vector) of node u in the adjacency matrix.

[0091] Step S65: Host Deletion. The host checks the value of pos. If it is the preset flag value, the update is terminated; otherwise, a logical deletion operation is performed on cols_vector[pos]. If the number of adjacent elements of node u is reduced to zero (cols_vector is empty), the corresponding adjacent matrix row is released.

[0092] Figure 8 and Figure 9 Together, they constitute the unloading-based graph update mechanism of this invention, realizing closed-loop collaboration between insertion and deletion in position management.

[0093] 3. Technical Effects Through the above mechanism, when updating the height number node, the host only needs to perform one O(1) memory write operation, while the complex edge search and space allocation logic is distributed to the parallel PIM module, achieving the optimization effect of fast processing of height number node updates.

[0094] VI. Node Dynamic Migration Strategy During system operation, the graph topology may change, causing the initial partitioning to fail. This invention performs dynamic adjustments through a node migration module, a process typically triggered after a query or update.

[0095] Degree-based migration: During the graph update phase, if a low-degree node on the PIM side exceeds a set degree threshold Th due to continuous edge insertion, the node migration module marks it and migrates its data and adjacency list from the PIM side to the host side, switching to a height-based node management mode. Conversely, if a height-based node on the host side falls below the threshold Th due to edge deletion, its data and adjacency list are migrated back to the PIM side, switching to a low-degree node management mode, thus achieving dynamic, bidirectional adjustment of node storage location.

[0096] Locality-based migration: During the query execution phase, the system calculates the "next-hop hit rate" of matched nodes in the PIM module. This rate represents the percentage of nodes whose next-hop neighbors are located within the same PIM module as the current node during the query execution process. For nodes with a next-hop hit rate lower than a preset percentage (e.g., below 25%, which can be dynamically configured by the system administrator based on actual workload), the system migrates the node to the target PIM module containing the most neighbors after the query is completed, thereby reducing cross-module communication in subsequent queries.

[0097] Furthermore, the other components and functions of the regular path query system based on in-memory computation in this embodiment of the invention are known to those skilled in the art, and will not be described in detail to reduce redundancy.

[0098] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. Unless otherwise specified, the embodiments and features described in this application can be arbitrarily combined with each other.

Claims

1. A regular expression path query system based on in-memory computation, characterized in that, include: Host side; The in-memory computing end contains multiple in-memory computing modules that work in parallel. The system divides the graph into non-overlapping nodes based on the degree characteristics of the graph nodes. Nodes with a degree exceeding a threshold and their adjacency relationships are allocated to the host for storage and processing, while nodes with a degree below the threshold and their adjacency relationships are allocated to the in-memory computing terminal for storage and processing.

2. The regular expression path query system based on in-memory computation according to claim 1, characterized in that, The system also includes a graph partitioning module for executing dynamic graph partitioning strategies: When a new node is inserted, the graph partitioning module obtains the neighbor node information of the new node in the current batch or in the historical cache; Identify the in-memory computing module containing the neighbor with the highest degree among the neighbor nodes; A multi-neighbor adaptive strategy is adopted to assign the new node to the in-memory computing module where the neighbor with the highest degree is located.

3. The regular path query system based on in-memory computation according to claim 2, characterized in that, The graph partitioning module is also used to perform load balancing: When allocating nodes, monitor the load of each in-memory computing module in real time; Set dynamic capacity constraints. When the load of the target in-memory computing module exceeds the constraint value calculated based on the average load of the entire system, prohibit the allocation of new nodes to the module, and reselect an in-memory computing module that is not overloaded and contains the neighbor of the new node for allocation.

4. The regular expression path query system based on in-memory computation according to claim 1, characterized in that, The system also includes a node migration module: During the graph data update process, nodes are migrated between the host and in-memory computing ends based on changes in node degree. During the path query process, monitor the next-hop hit rate of the nodes stored in the in-memory computing module, that is, the proportion of the next-hop neighbors located in the same in-memory computing module in the path query initiated by the node. For nodes whose next-hop hit rate is lower than a preset ratio, migrate them to the in-memory computing module that contains the most neighbors of that node.

5. The regular expression path query system based on in-memory computation according to claim 1, characterized in that, The host is configured to convert graph operations into a sequence of matrix-based operators and distribute them: Query transformation: The query module transforms regular path queries into a sequence of query operators that include matrix multiplication and reduction operators; Update Transformation: The graph update module transforms graph data changes into a sequence of update operators that include addition and subtraction operators; Operator execution: The in-memory computing module is equipped with an operator execution module, which is used to parse and execute the received query operator sequence and update operator sequence; Parallel filtering: The in-memory computing module is equipped with a filtering module, which is used to perform in-situ filtering on the path based on node labels or edge attributes in parallel during the stage of generating intermediate results by executing matrix multiplication operators, and only retain intermediate results that meet the filtering conditions to enter the next hop query.

6. The regular expression path query system based on in-memory computation according to claim 1, characterized in that, The system employs a heterogeneous graph storage mechanism: The host uses an adjacency matrix to store the next-hop neighbor list of the height node and allocates contiguous memory space for each height node. The in-memory computing terminal maintains an element position mapping table and a free position table for height number nodes in the in-memory computing module; When performing edge insertion or deletion operations on height nodes, the in-memory computing end uses the element position mapping table to query the specific position of the edge or uses the free position linked list to manage the available space, and returns the calculated position index to the host end. The host end only performs the write operation based on the position index.

7. A regular expression path query method based on in-memory computation, characterized in that, Includes the following steps: The storage process involves dividing the graph data into a set of nodes with high degree and a set of nodes with low degree, based on the degree of the graph nodes. The set of nodes with high degree is stored on the host side, while the set of nodes with low degree is distributed and stored in multiple in-memory computing modules on the in-memory computing side. Query distribution steps: The host receives a regular expression path query request and converts it into a matrix operation task; the operation tasks involving low-degree nodes are distributed to the corresponding in-memory computing modules for parallel execution, while the operation tasks involving high-degree nodes are executed on the host. Query execution steps: When performing path search, the in-memory computing module directly uses the locally stored attribute data to filter the generated intermediate results in situ according to the query rules; if the query does not reach the target number of hops and the next hop node is located in another in-memory computing module or on the host, then the intermediate results are exchanged. Once the target number of hops is reached, the partial query results will be returned to the host. Result reduction steps: After the host collects all local query results, it distributes these data to the in-memory computing module based on hash consistency to perform reduction in parallel, deduplicatizes and merges the results in parallel, and finally returns the normalized results to the host.

8. The regular expression path query method based on in-memory computation according to claim 7, characterized in that, The result reduction step includes: The host distributes each query result tuple to the corresponding in-memory computing module according to the hash consistency distribution rule; Each in-memory computation module maintains a hash set locally for deduplication. For each received tuple, it attempts to insert it into the hash set. If the insertion is successful, it is retained; if the insertion fails, it is discarded. After all in-memory computation modules have completed deduplication, they return all tuples in their respective hash sets to the host for merging, resulting in the final query result without duplicates.

9. The regular expression path query method based on in-memory computation according to claim 7, characterized in that, The method also includes a dynamic graph partitioning step: When inserting a new node, obtain the neighbor node information of the new node, and select the in-memory computing module containing the neighbor with the highest degree among the neighbors as the target module; Check if the current load of the target module exceeds the dynamic capacity constraint. If it does not exceed the constraint, assign the new node to the target module. If it does exceed the constraint, select the next in-memory computing module where the next neighbor is located in descending order of degree. If no neighboring module satisfies the constraints, the target module is selected from the set of in-memory computing modules whose load does not exceed the dynamic capacity constraints using a hash function.

10. The regular expression path query method based on in-memory computation according to claim 7, characterized in that, The method also includes a heterogeneous graph update step: When inserting edges into height nodes, the host distributes the edge's metadata to the corresponding in-memory computing module according to hash consistency. The in-memory calculation module queries the element position mapping table to determine if the edge exists. If it does not exist, it retrieves the free position index from the free position table, updates the element position mapping table, and returns the position index to the host. If the edge exists, the update is terminated. The host writes the target node into the contiguous array of the adjacency matrix according to the position index. When deleting an edge from a height node, the host distributes the edge's metadata to the corresponding in-memory computing module according to hash consistency. The in-memory calculation module queries the element position mapping table to obtain the position index, pushes the position index into the free position table, deletes the entry in the element position mapping table, and returns the position index to the host. The host performs logical deletion in the continuous array of the adjacency matrix according to the position index.

Images