Segmented aggregation hash table read-write method and system based on graph database

Abstract

The invention relates to a segmented aggregated hash table read-write method and system based on a graph database, and the method comprises the steps that an original hash table is separated into a hash entry table and a data load block, the hash entry table is used for storing hash entry data, and the data load block is used for storing aggregated data. And in response to the received write-in data instruction, positioning the hash entries in the hash entry table according to the hash value of the write-in data. Based on the data storage state of the hash entry, the write data is stored in the data load block through linear detection, and the hash entry is updated. And in response to receiving a data reading instruction, obtaining target aggregation data corresponding to the read data through linear detection according to the hash value of the read data. Therefore, the read-write performance of the hash table is improved, and the problem of low performance in the conventional read-write process of the hash table in a large data volume scene is solved.

Description

Technical Field

[0001] The present application relates to the field of databases, and in particular to a segmented aggregate hash table reading and writing method and system based on a graph database. Background Art

[0002] In the database field, hash tables, as the core data structure for implementing aggregate calculations, are widely used for efficient processing of grouped queries and aggregate functions (such as avg and count). Aggregate functions are more computationally complex than scalar functions. In scenarios with large amounts of data, queries involving aggregate calculations often suffer from poor performance, becoming a performance bottleneck for the entire business application.

[0003] Traditional hash tables achieve fast read and write speeds by mapping key-value pairs. Their performance relies on the uniform distribution of the hash function and its conflict resolution strategy. However, in scenarios with large amounts of data, frequent hash collisions can significantly degrade performance. For example, the unordered_map in the C++ Standard Template Library uses a linked list approach to handle collisions, attaching conflicting entries to hash buckets via pointers. While this design resolves conflicts, the dispersed storage of linked list nodes destroys data spatial locality, leading to inefficient CPU cache prefetching and, in turn, frequent cache misses and CPU stalls (which reduces pipeline efficiency and further impacts performance).

[0004] Therefore, in large data volume scenarios, the traditional hash table reading and writing process has the problem of low performance. Summary of the Invention

[0005] The embodiments of the present application provide a segmented aggregate hash table reading and writing method and system based on a graph database, so as to at least solve the problem of low performance in the traditional hash table reading and writing process in large data volume scenarios.

[0006] In a first aspect, an embodiment of the present application provides a segmented aggregate hash table reading and writing method based on a graph database, the method comprising: Separating the original hash table into a hash entry table and a data payload block, wherein the hash entry table is used to store hash entry data, and the data payload block is used to store aggregated data; In response to receiving a write data instruction, locating a hash entry in the hash entry table according to a hash value of the write data, Based on the data storage state of the hash entry, storing the write data in the data payload block by linear probing, and updating the hash entry; In response to receiving a read data instruction, target aggregate data corresponding to the read data is acquired through linear probing according to a hash value of the read data.

[0007] In one embodiment, the hash entry table is a list structure in a continuous memory layout, wherein the hash entry table includes multiple rows of hash entries, and each row of hash entries includes: Hash value prefix, used to store the high-order byte of the hash value; A block number, used to locate a target data load table in the data load block; An intra-block address, used to locate a data payload entry in the target data payload table.

[0008] In one embodiment, the data load block is a list structure in a continuous memory layout manner, and the data load block includes multiple data load tables with consistent structures. The data load table includes multiple rows of data load entries, and each row of the data load entries includes: Hash value, used to store the hash value of the grouping column. When writing data, the hash value of the grouping column is calculated and stored in advance; Grouping column, used to store grouping column data; Aggregate columns are used to store the calculation results of aggregate functions.

[0009] In one embodiment, locating a hash entry in the hash entry table according to the hash value of the written data includes: Calculating a hash value of the written data; Locating the row number of the write data in the hash entry table by taking the modulo of the hash value and the number of rows in the hash entry table; Based on the row number, a hash entry in the hash entry table is located.

[0010] In one embodiment, storing the write data in the data payload block by linear probing based on the data storage state of the hash entry and updating the data of the hash entry comprises the following steps: Determining whether the hash entry stores data; If the hash entry does not store data, the write data is stored in the data payload block and the data of the hash entry is updated; if the hash entry stores data, the write data is stored in the data payload block hash entry through linear probing.

[0011] In one embodiment, the write data is stored in the data payload block by linear probing, and the hash entry includes: Determine whether a hash prefix in the hash entry table of the next row matches a prefix of the hash value of the written data; If not, determining whether the hash prefix in the next row of the hash entry table matches the prefix of the hash value of the written data according to the continuous memory access rule; If there is a match, the corresponding data load entry is read according to the block number and the address within the block in the hash entry table, and it is determined whether the hash value in the data load entry is the same as the hash value of the written data, and whether the grouping column key value is the same as the key value of the written data. If they are the same, the aggregate column data corresponding to the grouping column key value of the data load block is updated.

[0012] In one embodiment, after updating the data of the hash entry, when the number of rows in the hash entry table reaches an expansion threshold, the method further includes: The capacity of the hash entry table and the data payload block is synchronously expanded to N times of the initial size; wherein the physical storage location in the data payload block remains unchanged, and the data in the hash entry table updates the hash entry through the hash value.

[0013] In one embodiment, in response to receiving the read data instruction, obtaining target aggregate data corresponding to the read data through linear probing according to the hash value of the read data includes: Calculate the hash value of the read data and perform linear detection in the hash entry; Determining whether a prefix of the hash value of the read data matches a hash prefix in the hash entry table; If there is no match, then according to the continuous memory access rule, determine whether the hash prefix in the hash entry table of the next row matches the prefix of the hash value of the read data; if they match, read the corresponding data load entry according to the block number and the address within the block in the hash entry table, and determine whether the hash value in the data load entry is the same as the hash value of the read data; if they are the same, determine whether the grouping column key value is the same as the key value of the read data; if they are the same, obtain the target aggregate data corresponding to the read data.

[0014] In one embodiment, after obtaining target aggregate data corresponding to the read data, the method further includes: Optimizing a segmented aggregate hash table based on data types, wherein, based on fixed-length data types, grouping column data of a data payload entry is stored at a hash prefix position of the hash entry, and based on variable-length string types, grouping columns are stored via a dictionary tree; And / or, optimizing the segmented aggregate hash table through a parallel computing strategy, partitioning the input data according to the hash range, processing each partition through an independent thread, each independent thread processing the hash entry table and the data load block, constructing a local hash table through address reference, and obtaining target aggregate data corresponding to the read data based on the local hash table.

[0015] In a second aspect, an embodiment of the present application provides a segmented aggregate hash table reading and writing system based on a graph database, the system comprising a separation module, a positioning module, a writing module, and a reading module; wherein: The separation module is used to separate the original hash table into a hash entry table and a data load block, wherein the hash entry table is used to store hash entry data, and the data load block is used to store aggregated data; The positioning module is configured to, in response to receiving a write data instruction, locate a hash entry in the hash entry table according to a hash value of the write data; The writing module is configured to store the write data in the data payload block through linear probing based on the data storage state of the hash entry, and update the hash entry; The reading module is configured to, in response to receiving a data reading instruction, obtain target aggregated data corresponding to the read data through linear probing according to a hash value of the read data.

[0016] In a third aspect, an embodiment of the present application provides a computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, the method for reading and writing a segmented aggregated hash table based on a graph database as described in the first aspect above is implemented.

[0017] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a segmented aggregate hash table reading and writing method based on a graph database as described in the first aspect above.

[0018] The embodiments of the present application provide a segmented aggregate hash table reading and writing method and system based on a graph database, which have at least the following technical effects.

[0019] The original hash table is separated into a hash entry table and a data payload block, wherein the hash entry table is used to store hash entry data, and the data payload block is used to store aggregate data. The hash entry table is decoupled from the data payload block to improve memory access locality and reduce frequent cache misses. In response to receiving a write data instruction, the hash entry in the hash entry table is located according to the hash value of the write data. The target entry is directly located by the hash value, avoiding the traversal process and reducing computational overhead. Based on the data storage status of the hash entry, the write data is stored in the data payload block through linear probing and the hash entry is updated. Conflicts are resolved in the continuous space of the hash entry table to accelerate the probing process. In response to receiving a read data instruction, the target aggregate data corresponding to the read data is obtained through linear probing according to the hash value of the read data. Invalid entries are quickly filtered through the hash prefix, reducing the number of accesses to the data payload block and improving the efficiency of aggregate data reading. When writing data, this application uses linear probing of the continuous memory layout to reduce cache failures and pre-stores hash values ​​to avoid repeated calculations. When querying data, most non-matching items are skipped through hash prefix layered filtering, and only a small amount of data load block data needs to be accessed, thereby improving the hash table read and write performance and solving the problem of low performance of the traditional hash table read and write process in large data volume scenarios.

[0020] The details of one or more embodiments of the present application are set forth in the following drawings and description to make other features, objects, and advantages of the present application more readily apparent. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings: Figure 1 It is a flowchart of a segmented aggregate hash table reading and writing method based on a graph database; Figure 2 is a schematic structural diagram of a hash entry table and a data load block according to an exemplary embodiment; Figure 3 is a flowchart of step S103 according to an exemplary embodiment; Figure 4 is a schematic structural diagram of a parallel segmented aggregation hash table according to an exemplary embodiment; Figure 5 This is a system structure diagram of segmented aggregate hash table reading and writing based on a graph database according to an exemplary embodiment; Figure 6 It is a structural block diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solutions and advantages of this application more clearly understood, the present application is described and illustrated below in conjunction with the accompanying drawings and examples. It should be understood that the specific embodiments described herein are merely used to explain this application and are not intended to limit this application. Based on the embodiments provided in this application, all other embodiments obtained by those of ordinary skill in the art without making any creative efforts are within the scope of protection of this application.

[0023] Obviously, the drawings described below are merely examples or embodiments of the present application. Those skilled in the art can, without inventive effort, apply the present application to other similar scenarios based on these drawings. Furthermore, it is also understood that, although the effort involved in such a development process may be complex and lengthy, for those skilled in the art related to the content disclosed in this application, changes in design, manufacturing, or production based on the technical content disclosed in this application are merely conventional technical means and should not be construed as an insufficiency of the content disclosed in this application.

[0024] References to "embodiments" in this application mean that a particular feature, structure, or characteristic described in connection with the embodiment may be included in at least one embodiment of the application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it refer to independent or alternative embodiments that are mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described in this application may be combined with other embodiments unless there is a conflict.

[0025] Unless otherwise defined, technical or scientific terms used herein shall have the ordinary meaning as understood by persons of ordinary skill in the art to which this application belongs. The terms "a," "an," "an," "the," and similar expressions used herein do not denote quantitative limitations and may refer to either the singular or the plural. The terms "comprise," "include," "have," and any variations thereof, used herein, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus comprising a series of steps or modules (units) is not limited to the listed steps or units but may also include steps or units not listed, or may include other steps or units inherent to the process, method, product, or apparatus. The terms "connected," "connected," "coupled," and similar expressions used herein are not limited to physical or mechanical connections but may include electrical connections, whether direct or indirect. As used herein, "plurality" means two or more. "And / or" describes an association between associated objects, indicating that three possible relationships exist. For example, "A and / or B" may mean: A exists alone; A and B exist simultaneously; or B exists alone. The character " / " generally indicates that the objects before and after are in an "or" relationship. The terms "first", "second", "third", etc. involved in this application are only used to distinguish similar objects and do not represent a specific order for the objects.

[0026] Aggregate functions are widely used in databases. Compared to scalar functions, they are computationally more complex. In large data scenarios, queries involving aggregate calculations often suffer from poor performance and are more likely to become performance bottlenecks for the entire business application. For example, a query statement first uses a graph topology to query all teachers and their students, then groups and aggregates each teacher by class and calculates the score statistics for each teacher's class. This query statement contains two aggregate functions: avg (the average score of students in the class) and count (the number of students who passed the class). This results in poor query performance.

[0027] In engineering practice, query performance involving aggregate functions in large data volumes is often poor, and query overhead is primarily concentrated in the aggregate operator. Performance analysis reveals that the performance bottleneck of the aggregate operator is primarily concentrated in the hash table overhead. For example: During the current data writing process, a traditional hash table first calculates the hash value of the key of the key-value pair to be written and locates it in a specific slot (bucket). If the slot is empty, the key-value pair is written to that slot. If it is not empty, a linked list node is created and connected to the end of the linked list of the current slot. Hash collisions occur when there are multiple key-value pairs within a slot. Hash collisions can severely degrade performance, with the worst-case time complexity dropping from O(1) to O(n). The solution to the problem of excessive hash collisions is to expand the capacity. This means that once the amount of data written exceeds a certain threshold, the hash table will be expanded. This expansion incurs the following overhead: a. Large amounts of data movement and copying b. Repeated hashing (rehashing) overhead.

[0028] During the existing read process, an unordered map first calculates the hash value of the query key and locates its slot in the hash table. Each slot in a traditional hash table stores all key-value pairs with the same hash value via a linked list. Once a slot is located, each key-value pair must be traversed to find pairs with the same key. This process incurs overhead: After locating the slot, all nodes within the slot must be traversed and their grouping columns compared for equality with the query key. In the worst case, all nodes within the slot must be traversed. The linked list data structure is implemented using pointers, which generates random memory accesses and reduces data locality, leading to CPU stalls. CPU stalls can severely degrade computing performance. Furthermore, when the hash table load factor exceeds a threshold, resizing requires recalculating the hash values ​​of all data and migrating memory. This involves extensive data copying and destruction (for automatic resource cleanup at the end of an object's lifecycle), further exacerbating write performance bottlenecks.

[0029] Therefore, in large data volume scenarios, traditional hash table read and write processes suffer from low performance. Improving the read and write performance of hash tables is crucial for aggregate computing performance.

[0030] Based on the above situation, an embodiment of the present application provides a segmented aggregate hash table reading and writing method and system based on a graph database.

[0031] It should be understood that the terms herein may be technical means for implementing a part of the present invention or other summary technical terms. For example, the terms may include: Database: Basic software used to manage data. The database provides data persistence and functions such as adding, deleting, modifying, and querying data. Users interact with the database through database languages ​​(such as SQL and GQL) or API interfaces to complete data processing business logic.

[0032] Aggregate function: A type of function in a database that aggregates multiple values ​​into one value. For example, the avg function calculates the average of multiple values, and the count function calculates the number of multiple values.

[0033] Aggregation operator: A type of operator in a database that groups input data and computes an aggregate function to produce the corresponding query result. The implementation of an aggregation operator has a significant impact on query performance.

[0034] Hash table: A key-value data structure in which data is stored as key-value pairs. A query can retrieve the corresponding key value using the key in O(1) time complexity. Each hash value in a hash table corresponds to a slot. If there are multiple key-value pairs within a slot, a hash collision will occur. This hash collision will severely degrade performance, with the worst-case time complexity dropping from O(1) to O(n). When the amount of data in a hash table exceeds a certain threshold, frequent hash collisions will inevitably occur, necessitating expansion (resize to double each time). This expansion requires recalculating hash values ​​for all data and rearranging their positions in the hash table based on the calculation results. This expansion operation involves a large amount of memory data movement, and the data movement overhead significantly impacts write performance. The unordered map in the C++ Standard Template Library is a common hash table implementation. In the database field, hash tables are often used to implement aggregation or join operators. Different scenarios may have their own specific optimization solutions. This application is suitable for aggregation computing scenarios.

[0035] Linear probing: A method for handling hash collisions. The main idea is to find the free unit closest to the conflicting unit in the hash table and write the data when a hash collision occurs.

[0036] Cache: The CPU's internal cache, which is faster than memory but more expensive to access. Cache leverages data locality to address the memory wall problem in von Neumann computers. A cache hit bypasses the expensive memory access instructions, allowing the CPU to access the cache instead, thus improving performance. Conversely, a cache miss not only results in a cache miss but also causes the CPU to stall (causing a pipeline bubble), resulting in reduced performance.

[0037] CPU stall: A CPU stall typically occurs due to a cache miss. After a cache miss, the CPU needs to execute memory access instructions, a process that consumes hundreds of CPU clock cycles without actually performing any work. On modern multi-core CPUs, CPU stalls can lead to reduced pipeline efficiency, significantly impacting performance.

[0038] In the first aspect, the embodiment of the present application provides a segmented aggregate hash table reading and writing method based on a graph database. Figure 1 This is a flowchart of a segmented aggregate hash table reading and writing method based on a graph database. Figure 1 As shown, the method includes: Step S101: Separate the original hash table into a hash entry table and a data payload block, wherein the hash entry table is used to store hash entry data, and the data payload block is used to store aggregated data.

[0039] Step S102 : In response to receiving a write data instruction, locate a hash entry in a hash entry table according to a hash value of the write data.

[0040] Step S103: Based on the data storage status of the hash entry, the write data is stored in the data payload block through linear probing, and the hash entry is updated.

[0041] Step S104 : in response to receiving the data read instruction, obtaining target aggregate data corresponding to the read data through linear probing according to the hash value of the read data.

[0042] In summary, an embodiment of the present application provides a segmented aggregate hash table reading and writing method based on a graph database, which separates the original hash table into a hash entry table and a data load block, wherein the hash entry table is used to store hash entry data, and the data load block is used to store aggregate data. The hash entry table is decoupled from the data load block to improve memory access locality and reduce frequent cache misses. In response to receiving a write data instruction, the hash entry in the hash entry table is located according to the hash value of the write data. The target entry is directly located by the hash value, avoiding the traversal process and reducing computational overhead. Based on the data storage status of the hash entry, the write data is stored in the data load block through linear probing, and the hash entry is updated. Conflicts are resolved in the continuous space of the hash entry table to accelerate the probing process. In response to receiving a read data instruction, the target aggregate data corresponding to the read data is obtained through linear probing according to the hash value of the read data. Invalid entries are quickly filtered through the hash prefix to reduce the number of accesses to the data load block and improve the efficiency of aggregate data reading. When writing, this application uses linear probing of the continuous memory layout to reduce cache failures, pre-stores hash values ​​to avoid repeated calculations, and when querying, skips most non-matching items through hash prefix layered filtering, requiring only access to a small amount of data payload block data, thereby improving the hash table read and write performance and solving the problem of low performance in the traditional hash table read and write process in large data volume scenarios.

[0043] Figure 2 FIG. 1 is a schematic diagram showing the structure of a hash entry table and a data load block according to an exemplary embodiment. Figure 2 As shown, step S101 is to separate the original hash table into a hash entry table and a data load block, wherein the hash entry table is used to store hash entry data and the data load block is used to store aggregated data. Specifically, it includes: The hash entry table is a list structure in a continuous memory layout, wherein the hash entry table includes multiple rows of hash entries, and each row of hash entries includes: Hash value prefix, used to store the high-order byte of the hash value; Block number, used to locate the target data load table in the data load block; The intra-block address used to locate the data payload entry in the target data payload table.

[0044] Optionally, the hash entry table uses a three-column structure to store the hash prefix (truncating the high-order bytes of the complete hash value), the block number, and the intra-block address. The aggregated data (including the grouping column and the aggregation column) in the data payload block is directly located through the block number and the intra-block address, avoiding the pointer jump of the linked list method. At the same time, the hash value of the grouping column is pre-stored in the data payload block header, so that when expanding the capacity, only the address mapping relationship of the hash entry table needs to be updated without migrating the data content, reducing the performance loss of data copying and destruction. In addition, hash prefix matching is used to accelerate conflict filtering, and combined with the prefetching characteristics of the continuous memory layout, invalid traversal and CPU downtime in the query phase are further reduced. The performance of the aggregation operator is improved by an order of magnitude compared to traditional hash tables, and is suitable for high-frequency aggregation computing scenarios such as databases and graph databases.

[0045] The data load block is a list structure with a continuous memory layout. The data load block includes multiple data load tables with the same structure. The data load table includes multiple rows of data load entries. Each row of data load entries includes: Hash value, used to store the hash value of the grouping column; Grouping column, used to store grouping column data; Aggregate columns are used to store the calculation results of aggregate functions.

[0046] Optionally, the data payload block stores multiple, structurally consistent data payload tables (DPTs) in a list structure with a contiguous memory layout. Each data payload entry (DE) in the data payload table contains the complete hash value of the grouping column, the grouping column data, and the aggregate column result. This has the following benefits: the contiguous memory layout accesses data via pointer offsets, avoiding the non-contiguous memory jumps of traditional linked lists, reducing cache misses and CPU stalls. The consistent structure of the data payload table design makes memory allocation and deallocation more efficient. The fixed-size data payload table also supports dynamic expansion by simply expanding the number of columns rather than migrating data content, reducing expansion overhead. Pre-stored hash values ​​for grouping columns can be used directly for fast hash collision verification and address mapping updates during expansion, avoiding the performance loss of repeated hash value calculations. The separate storage of grouping columns and aggregate columns allows aggregate calculation results (such as avg and count) to be directly updated in the aggregate column without reorganizing the original data, improving write efficiency. Furthermore, the prefetching feature of the contiguous memory accelerates data access during the query phase, reducing invalid traversals. This solves performance bottlenecks caused by hash collisions, expansion overhead, and cache misses when using large data volumes.

[0047] In one embodiment, step S102, in response to receiving a write data instruction, locates a hash entry in a hash entry table according to a hash value of the write data. Specifically, the step S102 includes: Calculate the hash value of the written data; The row number of the written data in the hash entry table is located by taking the remainder of the hash value and the number of rows in the hash entry table; Based on the row number, locate the hash entry in the hash entry table.

[0048] Optionally, the hash value is modulo the number of rows in the hash entry table and mapped to the physical address space of the hash table. For example, this achieves O(1) initial positioning efficiency, avoiding the non-contiguous memory jumps of traditional linked list methods. When locating hash entries based on row numbers and combining them with linear probing to handle conflicts, adjacent entries in contiguous memory are accessed sequentially, leveraging the CPU cache's prefetching mechanism to reduce cache misses and CPU stalls. While existing linked list methods require traversing non-contiguous memory nodes, this solution improves conflict handling efficiency by an order of magnitude through a continuous access pattern combining modulo positioning and linear probing. The design of pre-stored hash values ​​further accelerates the comparison process, ultimately reducing the overall performance loss of aggregate computing.

[0049] Figure 3 is a flowchart of step S103 according to an exemplary embodiment. Figure 3 As shown, step S103, based on the data storage status of the hash entry, the write data is stored in the data load block through linear detection, and the hash entry is updated. Specifically, the following steps are included: Step S1031: Determine whether the hash entry stores data; Step S1032: If the hash entry does not store data, the write data is stored in the data payload block and the data of the hash entry is updated. If the hash entry stores data, the write data is stored in the data payload block through linear detection.

[0050] Optionally, determining whether a hash entry stores data quickly identifies the validity of the initial positioning. If the entry is free, the write data is directly stored in the data payload block and the hash table entry is updated, avoiding unnecessary conflict handling. If a conflict exists, a linear probing method is used to sequentially search for adjacent entries and store the write data in the data payload block. Combined with the cache prefetching characteristics of the contiguous memory layout, this reduces cache misses and CPU stalls caused by random access. The hash entry data payload block avoids the pointer jump overhead of the linked list method, ultimately achieving high performance and low latency for write operations in aggregate computing scenarios.

[0051] It should be noted that data will only be written to the hash table entry when the corresponding entry row is empty after expansion or positioning. The data includes the hash prefix and the automatically assigned block number and address within the block; if the detection is not empty, the hash table entry does not need to be updated, that is, if the detection is not empty, the data does not need to be stored in the hash table entry.

[0052] Further, storing the write data in the data payload block by linear probing includes: Determine whether the hash prefix in the next row of hash entries matches the prefix of the hash value of the written data; If there is no match, then according to the continuous memory access rule, determine whether the hash prefix in the next row of hash entry table matches the prefix of the hash value of the written data; If there is a match, the corresponding data payload entry is read based on the block number and block address in the hash entry table. A determination is made as to whether the hash value in the data payload entry is the same as the hash value of the written data, and whether the grouping column key value is the same as the key value of the written data. If they are the same, the aggregate column data corresponding to the grouping column key value of the data payload block is updated. If they are not the same, the determination is continued as to whether the hash prefix in the next row of the hash entry table matches the prefix of the hash value of the written data. If they are the same, the aggregate column data corresponding to the grouping column key value of the data payload block is updated.

[0053] Optionally, hash prefix matching is used as the first-level filtering condition to quickly exclude non-target entries (such as conflicting entries not related to the written data) by intercepting the high-order bytes of the hash value, reducing unnecessary full hash value comparisons and grouping column key value verifications, thereby reducing the computational overhead of invalid operations; the continuous memory access rule utilizes the prefetch feature of the CPU cache to sequentially access adjacent hash table entries, avoiding the non-continuous memory jumps of the linked list method. When an entry with the same hash prefix is ​​matched, the full hash value and the grouping column key value are further verified to be completely consistent, ensuring data uniqueness. Redundant storage and calculation are avoided by directly updating the aggregate column instead of reorganizing the original data.

[0054] In one embodiment, after the data of the hash entry is updated, when the number of rows in the hash entry table reaches an expansion threshold, the following expansion operation is performed: The capacity of the hash entry table and the data payload block is expanded synchronously to N times of the initial size; wherein the physical storage location in the data payload block remains unchanged, and the data in the hash entry table updates the hash entry through the hash value.

[0055] Optionally, when the hash table load factor exceeds a threshold, the expansion operation requires recalculating the hash values ​​of all data and migrating memory, involving a large amount of data copying and destruction operations, further exacerbating the write performance bottleneck. Therefore, when the number of rows in the hash entry table reaches the expansion threshold, the capacity of the hash table and the data payload block is synchronously expanded to N times the initial size. In the embodiment of the present application, this can be 1-2 times. This avoids the high overhead of full data migration and repeated hash calculations during traditional hash table expansion. Specifically, the physical storage location of the data payload block remains unchanged, and expansion is achieved only by expanding its capacity (such as appending a fixed-size data payload table), without migrating the original aggregated data content. This avoids the data copying and destruction performance loss associated with traditional linked list or array expansion. At the same time, the hash entry updates the address mapping using pre-stored hash values, and the row number of the new hash table is directly calculated using the complete hash value of the grouping column, avoiding the need to recalculate the hash values ​​of all entries and significantly reducing the computational overhead during expansion.

[0056] In one embodiment, step S104, in response to receiving a data read instruction, obtains target aggregate data corresponding to the read data through linear probing according to the hash value of the read data. Specifically, it includes: Calculate the hash value of the read data and perform linear probing in the hash entries; Determine whether the prefix of the hash value of the read data matches the hash prefix in the hash entry table; If there is no match, the continuous memory access rule is used to determine whether the hash prefix in the next row of hash entries matches the prefix of the hash value of the read data. If so, the corresponding data payload entry is read based on the block number and block address in the hash entry table. The hash value in the data payload entry is determined to be the same as the hash value of the read data. If they are the same, the grouping column key value is determined to be the same as the key value of the read data. If they are the same, the target aggregate data corresponding to the read data is obtained. If they are not the same, the linear detection is continued in the hash entries until the target aggregate data corresponding to the read data is obtained.

[0057] Optionally, when reading data, hash prefix matching serves as the first level of filtering to quickly exclude non-target entries (such as conflicting entries unrelated to the data being read), reducing unnecessary full hash value comparisons and grouping column key value verifications, thereby reducing the computational overhead of invalid operations. Linear probing, combined with continuous memory access rules, sequentially accesses adjacent hash table entries, leveraging the prefetching capabilities of the CPU cache to reduce cache misses and CPU stalls caused by random access. When an entry with a matching hash prefix is ​​matched, the full hash value and grouping column key value are further verified to ensure complete consistency, ensuring data uniqueness. Aggregate data is directly located using block numbers and intra-block addresses, avoiding the pointer jump overhead of linked list methods.

[0058] In one embodiment, after obtaining target aggregate data corresponding to the read data, the method further includes: The segmented aggregate hash table is optimized according to the data type, wherein, according to the fixed-length data type, the grouping column data of the data load entry is stored in the hash prefix position of the hash entry, and according to the variable-length string type, the grouping column is stored through the dictionary tree.

[0059] Optionally, for different types of aggregated data, different memory solutions can be designed for different data types due to their different memory space occupation and data type characteristics. Fixed-length data type (integer optimization): Taking int16 as an example, the grouping column only occupies 2 bytes. To avoid non-contiguous memory access from the hash entry table to the data payload block, the int16 type grouping column data can be moved from the data payload entry to the hash prefix position of the hash entry. During linear probing, key equality determination can be performed directly in the hash entry table. String type optimization: The string type is a variable-length type and its memory space occupation needs to be determined based on the string length. However, the string data type characteristic is that all data consists of enumerable characters, and memory space can be saved by constructing a dictionary tree.

[0060] And / or, optimizing the segmented aggregate hash table through a parallel computing strategy, partitioning the input data according to the hash range, processing each partition through an independent thread, each independent thread processing the hash entry table and the data load block, building a local hash table through address reference, and obtaining the target aggregate data corresponding to the read data based on the local hash table.

[0061] Optionally, Figure 4 FIG. 1 is a schematic diagram showing a structure of a parallel segmented aggregation hash table according to an exemplary embodiment. Figure 4 As shown, the input data received by the aggregation operator is first partitioned, and different partitioned data are sent to different independent threads for processing. Each independent thread processes the hash entry table and data load block, providing users with a local hash table (such as Figure 4 Based on the local hash tables, a global hash table view is obtained, and based on the global hash table view, the target aggregate data corresponding to the read data is obtained. The global hash table view does not copy the actual data but only references the actual data by address.

[0062] In summary, an embodiment of the present application provides a segmented aggregate hash table reading and writing method based on a graph database, which separates the original hash table into a hash entry table and a data load block, wherein the hash entry table is used to store hash entry data, and the data load block is used to store aggregate data. The hash entry table is decoupled from the data load block to improve memory access locality and reduce frequent cache misses. In response to receiving a write data instruction, the hash entry in the hash entry table is located according to the hash value of the write data. The target entry is directly located by the hash value, avoiding the traversal process and reducing computational overhead. Based on the data storage status of the hash entry, the write data is stored in the data load block through linear probing, and the hash entry is updated. Conflicts are resolved in the continuous space of the hash entry table to accelerate the probing process. In response to receiving a read data instruction, the target aggregate data corresponding to the read data is obtained through linear probing according to the hash value of the read data. Invalid entries are quickly filtered through the hash prefix to reduce the number of accesses to the data load block and improve the efficiency of aggregate data reading. When writing, this application uses linear probing of the continuous memory layout to reduce cache failures, pre-stores hash values ​​to avoid repeated calculations, and when querying, skips most non-matching items through hash prefix layered filtering, requiring only access to a small amount of data payload block data, thereby improving the hash table read and write performance and solving the problem of low performance in the traditional hash table read and write process in large data volume scenarios.

[0063] In a second aspect, an embodiment of the present application provides a segmented aggregate hash table reading and writing system based on a graph database. Figure 5 This is a system structure diagram of a segmented aggregate hash table reading and writing based on a graph database according to an exemplary embodiment. Figure 5 As shown, the system includes a separation module 510, a positioning module 520, a writing module 530 and a reading module 540; wherein: a separation module 510, configured to separate the original hash table into a hash entry table and a data payload block, wherein the hash entry table is used to store hash entry data, and the data payload block is used to store aggregated data; The positioning module 520 is configured to locate a hash entry in the hash entry table according to the hash value of the written data in response to receiving a write data instruction. A write module 530 for storing write data in a data payload block through linear probing based on a data storage state of the hash entry and updating the hash entry; The reading module 540 is configured to, in response to receiving a read data instruction, obtain target aggregate data corresponding to the read data through linear probing according to a hash value of the read data.

[0064] In summary, the present application provides a segmented aggregate hash table read-write system based on a graph database, which includes a separation module 510, a positioning module 520, a write module 530, and a read module 540. During writing, linear probing of a continuous memory layout reduces cache invalidation, pre-stored hash values ​​avoid repeated calculations, and during querying, most non-matching items are skipped through hash prefix hierarchical filtering, requiring only access to a small amount of data payload block data, thereby improving hash table read and write performance and solving the problem of low performance in traditional hash table read and write processes in large data volume scenarios.

[0065] It should be noted that this embodiment provides a segmented aggregate hash table reading and writing system based on a graph database for implementing the above-mentioned implementation mode, and the details that have been explained will not be repeated here. As used above, the terms "module", "unit", "sub-unit", etc. can be a combination of software and / or hardware that implements the predetermined functions. Although the apparatus described in the above embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceived.

[0066] In a third aspect, an embodiment of the present application provides an electronic device, Figure 6 FIG is a block diagram of an electronic device according to an exemplary embodiment. Figure 6 As shown, the electronic device may include a processor 61 and a memory 62 storing computer program instructions.

[0067] Specifically, the processor 61 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0068] The memory 62 may include a large-capacity memory for data or instructions. By way of example, and not limitation, the memory 62 may include a hard disk drive (HDD), a floppy disk drive, a solid-state drive (SSD), flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, the memory 62 may include removable or non-removable (or fixed) media. Where appropriate, the memory 62 may be internal or external to the data processing device. In certain embodiments, the memory 62 is non-volatile memory. In certain embodiments, the memory 62 includes read-only memory (ROM) and random access memory (RAM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically alterable ROM (EAROM) or a flash memory (FLASH), or a combination of two or more of these. Under appropriate circumstances, the RAM can be a static random access memory (SRAM) or a dynamic random access memory (DRAM), where the DRAM can be a fast page mode dynamic random access memory (FPMDRAM), an extended data out dynamic random access memory (EDODRAM), a synchronous dynamic random access memory (SDRAM), etc.

[0069] The memory 62 may be used to store or cache various data files required for processing and / or communication, as well as possible computer program instructions executed by the processor 61 .

[0070] The processor 61 implements any one of the segmented aggregate hash table reading and writing methods based on a graph database in the above embodiments by reading and executing computer program instructions stored in the memory 62 .

[0071] In one embodiment, the segmented aggregate hash table reading and writing device based on the graph database may further include a communication interface 63 and a bus 60. Figure 6 As shown, the processor 61, the memory 62, and the communication interface 63 are connected via a bus 60 and communicate with each other.

[0072] The communication interface 63 is used to enable communication between the various modules, devices, units, and / or devices in the embodiments of the present application. The communication port 63 can also enable data communication with other components such as external devices, image / data acquisition devices, databases, external storage, and image / data processing workstations.

[0073] Bus 60, comprising hardware, software, or both, couples the components of the segmented aggregate hash table read / write device based on the graph database. Bus 60 includes, but is not limited to, at least one of the following: a data bus, an address bus, a control bus, an expansion bus, and a local bus. By way of example and not limitation, bus 60 may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an InfiniBand interconnect, a Low Pin Count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local Bus (VLB) bus, or other suitable buses, or a combination of two or more of these. Bus 60 may include one or more buses, where appropriate. Although embodiments herein describe and illustrate a particular bus, this application contemplates any suitable bus or interconnect.

[0074] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a program stored thereon. When the program is executed by a processor, the segmented aggregate hash table reading and writing method based on a graph database provided in the first aspect is implemented.

[0075] The readable storage medium may include, but is not limited to, a portable disk, a hard disk, a random access memory, a read-only memory, an erasable programmable read-only memory, an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0076] In a possible embodiment, the present invention can also be implemented in the form of a program product, which includes program code. When the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps of the segmented aggregate hash table reading and writing method based on the graph database provided in the first aspect.

[0077] The program code for executing the present invention may be written in any combination of one or more programming languages, and may be executed entirely on the user device, partially on the user device, as an independent software package, partially on the user device and partially on a remote device, or entirely on the remote device.

[0078] The technical features of the above-mentioned embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above-mentioned embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0079] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. A segmented aggregate hash table reading and writing method based on a graph database, characterized in that: The method comprises: Separating the original hash table into a hash entry table and a data payload block, wherein the hash entry table is used to store hash entry data, and the data payload block is used to store aggregated data; In response to receiving a write data instruction, locating a hash entry in the hash entry table according to a hash value of the write data; Based on the data storage state of the hash entry, storing the write data in the data payload block by linear probing, and updating the hash entry; In response to receiving a read data instruction, target aggregate data corresponding to the read data is acquired through linear probing according to a hash value of the read data.

2. A segmented aggregate hash table reading and writing method based on a graph database according to claim 1, characterized in that: The method of storing the write data in the data load block by linear probing based on the data storage state of the hash entry and updating the data of the hash entry comprises the following steps: Determining whether the hash entry stores data; If the hash entry does not store data, the write data is stored in the data payload block and the data of the hash entry is updated; if the hash entry stores data, the write data is stored in the data payload block through linear probing.

3. A segmented aggregate hash table reading and writing method based on a graph database according to claim 2, characterized in that: The step of storing the write data in the data load block by linear probing includes: Determine whether a hash prefix in the hash entry table of the next row matches a prefix of the hash value of the written data; If not, determining whether the hash prefix in the next row of the hash entry table matches the prefix of the hash value of the written data according to the continuous memory access rule; If there is a match, the corresponding data load entry is read according to the block number and the address within the block in the hash entry table, and it is determined whether the hash value in the data load entry is the same as the hash value of the written data, and whether the grouping column key value in the data load entry is the same as the key value of the written data. If they are the same, the aggregate column data corresponding to the grouping column key value of the data load block is updated.

4. A segmented aggregate hash table reading and writing method based on a graph database according to claim 2, characterized in that: After updating the data of the hash entry, when the number of rows in the hash entry table reaches an expansion threshold, the method further includes: The capacity of the hash entry table and the data payload block is synchronously expanded to N times of the initial size; wherein the physical storage location in the data payload block remains unchanged, and the data in the hash entry table updates the hash entry through the hash value.

5. The segmented aggregate hash table reading and writing method based on a graph database according to claim 1 is characterized in that: The method of acquiring target aggregate data corresponding to the read data by linear probing according to a hash value of the read data in response to receiving the read data instruction includes: Calculate the hash value of the read data and perform linear detection in the hash entry; Determining whether a prefix of the hash value of the read data matches a hash prefix in the hash entry table; If there is no match, then according to the continuous memory access rule, determine whether the hash prefix in the hash entry table of the next row matches the prefix of the hash value of the read data; if they match, read the corresponding data load entry according to the block number and the address within the block in the hash entry table, and determine whether the hash value in the data load entry is the same as the hash value of the read data; if they are the same, determine whether the grouping column key value in the data load entry is the same as the key value of the read data; if they are the same, obtain the target aggregate data corresponding to the read data.

6. A segmented aggregate hash table reading and writing method based on a graph database according to claim 1, characterized in that: The hash entry table is a list structure in a continuous memory layout manner, wherein the hash entry table includes multiple rows of hash entries, and each row of hash entries includes: Hash value prefix, used to store the high-order byte of the hash value; A block number, used to locate a target data load table in the data load block; An intra-block address, used to locate a data payload entry in the target data payload table.

7. A segmented aggregate hash table reading and writing method based on a graph database according to claim 1, characterized in that: The data load block is a list structure in a continuous memory layout mode. The data load block includes multiple data load tables with consistent structures. The data load tables include multiple rows of data load entries. Each row of the data load entries includes: Hash value, used to store the hash value of the grouping column, the hash value of the grouping column is calculated when writing data and pre-stored in the data payload entry; Grouping column, used to store grouping column data; Aggregate columns are used to store the calculation results of aggregate functions.

8. The segmented aggregate hash table reading and writing method based on a graph database according to claim 1 is characterized in that: The locating a hash entry in the hash entry table according to the hash value of the written data includes: Calculating a hash value of the written data; Locating the row number of the write data in the hash entry table by taking the modulo of the hash value and the number of rows in the hash entry table; Based on the row number, a hash entry in the hash entry table is located.

9. A segmented aggregate hash table reading and writing method based on a graph database according to any one of claims 1 to 8, characterized in that: After acquiring target aggregate data corresponding to the read data, the method further includes: Optimizing a segmented aggregate hash table based on data types, wherein, based on fixed-length data types, grouping column data of a data payload entry is stored at a hash prefix position of the hash entry, and based on variable-length string types, grouping columns are stored via a dictionary tree; And / or, optimizing the segmented aggregate hash table through a parallel computing strategy, partitioning the input data according to the hash range, processing each partition through an independent thread, each independent thread processing the hash entry table and the data load block, constructing a local hash table through address reference, and obtaining target aggregate data corresponding to the read data based on the local hash table.

10. A segmented aggregate hash table reading and writing system based on a graph database, characterized in that: The system includes a separation module, a positioning module, a writing module and a reading module; wherein: The separation module is used to separate the original hash table into a hash entry table and a data load block, wherein the hash entry table is used to store hash entry data, and the data load block is used to store aggregated data; The positioning module is configured to, in response to receiving a write data instruction, locate a hash entry in the hash entry table according to a hash value of the write data; The writing module is configured to store the write data in the data payload block through linear probing based on the data storage state of the hash entry, and update the hash entry; The reading module is configured to, in response to receiving a data reading instruction, obtain target aggregated data corresponding to the read data through linear probing according to a hash value of the read data.

Images