First-in first-out data structure and processing method based on key value pair search

Abstract

The invention discloses a first-in first-out data structure based on key value pair search and a processing method, relates to data processing and data storage technologies, and comprises a three-layer closed-loop architecture controlled by a hash index, an annular array and metadata. Wherein the Hash index layer is used for receiving an external key value operation request and positioning a data position in an annular array through index mapping; the annular array storage layer is used for physical storage and sequence management of data, an annular array of a continuous memory is adopted, and FIFO sequence management is achieved through a compact data unit and head and tail pointers; and the metadata control layer is used for maintaining the running state of the structure and providing parameter support for operation execution, and the metadata control layer adopts atomic variables to store core state parameters and realizes lock-free synchronization through CAS operation. The storage structure is optimized, the precursor / subsequent pointers of the double linked list are cancelled, a compact storage mode is adopted, and the memory utilization rate in a small data scene is improved.

Description

Technical Field

[0001] This application relates to the fields of data processing and data storage technology, and in particular to a first-in-first-out (FIFO) data structure and processing method based on key-value pair lookup. Background Technology

[0002] With the rapid deployment of the Internet of Things (IoT) and edge computing technologies, data processing scenarios are shifting from the cloud to edge nodes (such as edge gateways, industrial sensors, and smart terminals). These devices exhibit significant scenario characteristics: limited resources, with memory typically ranging from a few MB to tens of MB, resulting in weak computing power; prominent data characteristics, generating real-time data at high frequency, with queries concentrated on key-value mappings, and the need to evict expired data in chronological order; and stringent low-latency requirements, with query response times needing to be controlled below milliseconds. For example, industrial sensors can generate thousands of device status data points per second (the key being the device ID / data number, and the value being the monitoring indicator). Edge gateways need to quickly query data from specific devices and, when memory is insufficient, evict the oldest data according to a "first-in, first-out (FIFO)" rule to avoid memory overflow.

[0003] The well-known underlying technologies supporting data processing in this scenario mainly consist of two types of traditional data structures: FIFO queues: These use a linear storage method with arrays or singly linked lists, and data is dequeued in the order it was enqueued, naturally achieving strict FIFO eviction. However, the search operation requires traversing all elements, with a time complexity of O(n). When the amount of data reaches 1000 records, the search latency usually exceeds 10ms, which cannot meet the needs of high-frequency queries.

[0004] Hash Tables: Based on hash functions, hash tables establish a direct mapping between keys and values. Search, insertion, and deletion operations all have a time complexity of O(1), allowing for fast response to key-value queries. However, they lack sequential management capabilities, cannot implement FIFO (First-In, First-Out) eviction, and continuous data writing can easily lead to memory exhaustion. Additional auxiliary structures such as timestamps and index tables need to be maintained, increasing system complexity and memory overhead.

[0005] In existing technologies, the hybrid structure of hash table + doubly linked list (typically represented by Java LinkedHashMap) is the mainstream solution that balances "key-value pair lookup" and "FIFO eviction". Specifically, this solution includes two core components that work together: Hash table: It uses an array as the underlying storage container. Each array element (hash bucket) corresponds to a set of key-value pairs with the same hash value. Through the mapping relationship of "key-linked list node", it can realize the fast location of data and support the search operation with O(1) time complexity.

[0006] Doubly linked list: Each node contains four parts: "key, value, previous pointer, and next pointer". The previous / next pointers are used to maintain the order relationship between nodes, so that the linked list is sorted according to the data insertion time, providing the order basis for FIFO elimination.

[0007] The data insertion operation calculates the hash value of the "key" of the data to be inserted and determines its position in the target hash bucket of the hash table; Check if there are key collisions in the hash bucket: if there are no collisions, create a new doubly linked list node and insert it into the hash bucket; if there are collisions, attach the new node to the corresponding position according to the hash table collision resolution strategy (such as chaining). At the same time, insert the new node as the tail node of the doubly linked list, update the "next" pointer of the original tail node to point to the new node, and the "prev" pointer of the new node to point to the original tail node, thus completing the sequential update of the linked list. Record the mapping relationship between "key-new linked list node" in the hash table to ensure that the node can be quickly located by key in the future.

[0008] The data lookup operation calculates the hash value of the "key" of the data to be searched and locates the target hash bucket in the hash table; Traverse the nodes within the hash bucket (resolving key collisions) and find the corresponding doubly linked list node through "key matching"; The value stored in the node can be returned directly without traversing the doubly linked list, achieving a search time complexity of O(1).

[0009] The FIFO elimination operation pre-sets the data storage limit (such as the maximum number of nodes) and monitors the number of nodes in the doubly linked list in real time. When the number of nodes reaches the storage limit, an eviction mechanism is triggered: the head node of the doubly linked list (i.e., the earliest inserted data node) is determined. Perform a deletion operation: Modify the "prev" pointer of the successor node of the head node to point to null, and disconnect it from the head node; at the same time, delete the mapping entry of the "key" corresponding to the head node in the hash table. Release the memory occupied by the head node, complete a single FIFO eviction, and ensure that the total amount of data does not exceed the storage limit.

[0010] In addition, there is a variant of this scheme that uses a "ring buffer + hash table", but the overwrite mechanism of the ring buffer is prone to breaking the strict FIFO property, and the synchronous update overhead of the "key-buffer position" mapping is large.

[0011] The existing "circular buffer + hash table" technology requires doubly linked list nodes to additionally store "prev" and "next" pointers—each pointer occupies 4 bytes in a 32-bit system and 8 bytes in a 64-bit system. However, in IoT scenarios, over 80% of the data is small-sized (such as 8-byte sensor status data or 16-byte device ID mapping data). Pointers consume over 50% of memory (e.g., in a 64-bit system, a node with 16 bytes of data requires an additional 16-byte pointer, doubling the total memory usage). The limited memory of edge devices, ranging from a few MB to tens of MB, is consumed by a large amount of non-functional pointer overhead, directly reducing the amount of data that can be stored by 30%-50%, making it unsuitable for resource-constrained environments.

[0012] Insertion and deletion operations require the simultaneous maintenance of a hash table and a doubly linked list. Insertion involves four steps: calculating the hash value, inserting into the hash bucket, modifying the pointer of the tail node in the linked list, and updating the pointer of the new node. Deletion involves four steps: locating the hash bucket, finding the linked list node, modifying the pointers of the previous and current nodes, and deleting the hash table mapping. These multiple pointer modifications and component synchronization consume significant CPU resources. In edge device scenarios with tens of thousands of data reads and writes per second, CPU utilization increases from the usual 20% to over 60%, and single-operation response latency increases from milliseconds (<1ms) to tens of milliseconds (>10ms), failing to meet the low-latency requirements of high-frequency data processing in the Internet of Things (IoT).

[0013] FIFO eviction must be executed according to the "single-node deletion" logic—each eviction can only delete the head node of the doubly linked list, and the process of "modifying pointers → deleting hash mappings → releasing memory" must be repeated. Evicting N data items requires performing N complete deletion operations, and the eviction efficiency is linearly negatively correlated with the amount of data. When edge devices encounter sudden data peaks (such as tens of thousands of temporary data items generated per second), and thousands of the oldest data items need to be evictioned, the total eviction time exceeds 1 second. Memory cannot be released quickly, which can easily lead to memory overflow and data loss, making it unable to cope with sudden demands in the scenario.

[0014] Doubly linked list nodes employ a discrete storage model—each node requires its own memory space, and the node size (data + pointer) varies with the data type. Frequent "node creation → deletion" operations by edge devices generate numerous small, discontinuous free blocks in memory. This memory fragmentation rate reaches 20%-30%. Even if the total free memory meets the demand, it's impossible to allocate contiguous space to store new nodes, forcing premature eviction mechanisms and further compressing the amount of storable data, exacerbating resource waste.

[0015] Traditional FIFO queues can only perform linear traversal searches (O(n) complexity), while hash tables lack sequential management capabilities. Existing hybrid structures, while combining hash tables and doubly linked lists, are essentially "splitting together two independent components," failing to optimize the underlying coordination logic and still incurring synchronization overhead between components. In scenarios where high-frequency searches and high-frequency evictions coexist (such as real-time monitoring of industrial sensors), this synchronization overhead leads to either "fast searches but slow evictions" or "increased search latency during eviction adjustments," failing to achieve seamless coordination between the two and exhibiting insufficient adaptability. Summary of the Invention

[0016] This application provides a first-in-first-out (FIFO) data structure and processing method based on key-value pair lookup. By optimizing the storage structure, eliminating the predecessor / successor pointers of the doubly linked list, and adopting a compact storage mode, it improves memory utilization in small data scenarios, solves the problem of excessive memory consumption caused by pointer redundancy, and adapts to the resource-constrained characteristics of edge devices.

[0017] This application provides a first-in, first-out (FIFO) data structure based on key-value pair lookup, comprising a three-layer closed-loop architecture of hash index, circular array, and metadata control, wherein each layer collaborates through a closed-loop mechanism of index mapping, data storage, and state management; wherein, The hash index layer is used to receive external key-value operation requests and locate the data position in the circular array through index mapping. The hash index layer adopts a storage design of hash digest and index. The circular array storage layer is used for physical storage and sequential management of data. It adopts a circular array with contiguous memory and achieves FIFO sequential management through compact data units and head and tail pointers. The metadata control layer maintains the structure's operating state and provides parameter support for operation execution. The metadata control layer uses atomic variables to store core state parameters and achieves lock-free synchronization through CAS operations.

[0018] This application provides a first-in-first-out (FIFO) data processing method based on key-value pair lookup, implemented using the aforementioned FIFO data structure, including: Receive external key-value operation requests and locate the data position in the circular array through index mapping; A circular array with contiguous memory is used to manage the physical storage and order of data through compact data units and head and tail pointers; The core state parameters are stored using atomic variables, and lock-free synchronization is achieved through CAS operations to maintain the running state of the data structure.

[0019] This application's embodiments optimize the storage structure, eliminate the predecessor / successor pointers of the doubly linked list, adopt a compact storage mode, improve memory utilization in small data scenarios, solve the problem of excessive memory consumption caused by pointer redundancy, and adapt to the resource-constrained characteristics of edge devices.

[0020] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, the following are specific embodiments of this application. Attached Figure Description

[0021] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings: Figure 1 This application presents a three-layer collaborative architecture based on a key-value pair lookup-based first-in-first-out data structure. Figure 2 This is a schematic diagram of the first-in-first-out data insertion operation based on key-value pair lookup in an embodiment of this application; Figure 3 This is a schematic diagram of the first-in-first-out (FIFO) data key-value lookup operation flow based on key-value pair lookup in an embodiment of this application; Figure 4 This is a schematic diagram of the first-in-first-out (FIFO) data eviction operation based on key-value pair lookup, as described in an embodiment of this application. Detailed Implementation

[0022] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0023] This application provides a first-in-first-out (FIFO) data structure based on key-value pair lookup, such as... Figure 1As shown, a three-layer closed-loop architecture including hash index, circular array, and metadata control is presented. Each layer collaborates through a closed loop of index mapping, data storage, and state management. This application pioneers a three-layer closed-loop architecture of "hash index - circular array - metadata control," breaking the traditional component separation model of "hash table + doubly linked list." This architecture organically integrates key-value positioning, data storage, and state management, achieving for the first time a native unification of O(1) lookup performance and a strict FIFO elimination mechanism, completely eliminating the additional overhead caused by component synchronization. The hash index layer is used to receive external key-value operation requests and locate data positions in the circular array through index mapping. The hash index layer adopts a storage design of hash digest and index. In some embodiments of this application, each index entry in the hash index layer includes a hash digest of a specified number of bits, a circular array index, and a status identifier, and the total length does not exceed 64 bits, wherein the status identifier is used to mark the validity of the entry and is 8 bits long.

[0024] In one specific example, each index entry includes a "32-bit key hash digest", a "16-bit circular array index", and an "8-bit status flag", with a total length of only 64 bits (8 bytes). This example reduces memory usage by 75% compared to traditional hash tables that store complete key-value pairs (at least 32 bytes). The status flag is used to mark the validity of the entry (0 - valid, 1 - deleted, 2 - conflict), avoiding the "tombstone" marking overhead of traditional open addressing.

[0025] For hash algorithm selection, this example uses the MurmurHash3 (non-cryptographic hash function) algorithm, which has a hash collision rate of less than 0.01% for common key types such as device ID and sensor number, and the algorithm execution time is only 0.02ms, making it suitable for the limited computing power of edge devices. In the 64-bit hash value output by the algorithm, the high 32 bits are stored as the key hash digest, and the low 16 bits are used as the initial index position, reducing the index calculation time.

[0026] The circular array storage layer is used for physical storage and sequential management of data. It adopts a circular array of contiguous memory and implements FIFO sequential management through compact data units and head and tail pointers.

[0027] The metadata control layer maintains the structure's operating state and provides parameter support for operation execution. The metadata control layer uses atomic variables to store core state parameters and achieves lock-free synchronization through CAS operations.

[0028] This application's embodiments optimize the storage structure, eliminate the predecessor / successor pointers of the doubly linked list, adopt a compact storage mode, improve memory utilization in small data scenarios, solve the problem of excessive memory consumption caused by pointer redundancy, and adapt to the resource-constrained characteristics of edge devices.

[0029] In some embodiments of this application, the hash index layer adopts a hybrid approach of linear probing and collision chaining. In the case of initial index position collision, free positions are probed in the order of index + 1. Furthermore, collision chains are recorded through status identifiers. Subsequent searches only need to traverse the collision chains, avoiding full table scans. In collision scenarios, the search time is controlled within 0.1ms.

[0030] In some embodiments of this application, the structural parameters of the circular array in the circular array storage layer are: The initial length is 2^n. The array uses contiguous memory allocation. Each expansion doubles the original length. During expansion, valid data is migrated in batches using a memory copy function. For example, the initial length is set to 2^10 (1024).

[0031] Each of the compact data units in the circular array storage layer includes: The Key ID is consistent with the length of the hash index layer, based on the 64-bit Key ID of the previous example.

[0032] The Value field uses a length prefix encoding, which includes a 1-byte length identifier and the data content, and supports various data types from 8-byte sensor data to 128-byte device logs.

[0033] Timestamps are used to help verify the validity of data.

[0034] In some embodiments of this application, the head and tail pointers of the circular array storage layer include a head pointer and a tail pointer to control data input and output, wherein, The head pointer points to the earliest inserted data position; The tail pointer points to the position to be inserted. If the array is not full, the tail pointer moves by Tail=(Tail+1)%Length, where Length is the length of the circular array; When the array is full, new data overwrites the head pointer position, and the head pointer moves synchronously.

[0035] In some embodiments of this application, the core metadata of the metadata control layer includes the length of the circular array, the current data volume, the position of the head pointer, the position of the tail pointer, and the hash index layer load factor, such as 0.7. All core metadata is stored with the same length as the hash index layer, for example, using 64-bit atomic variables as described in the previous examples, supporting atomic read and write operations. This application uses a contiguous memory circular array as a carrier, designs a pointerless data unit structure to reduce memory fragmentation, and achieves natural FIFO elimination through atomic movement of the head / tail pointers. The application's 2x expansion + memcpy batch migration mechanism increases memory utilization to over 90% and reduces fragmentation to below 10%, solving the memory waste problem of traditional storage structures.

[0036] In some embodiments of this application, during insertion / deletion operations, the Head / Tail pointers and Count value are atomically updated using CAS operations. For example, during an insertion operation, the Tail position is first obtained via CAS, data is written, and then the Tail value is updated via CAS. If the update fails, it is retried (the number of retries does not exceed 3), ensuring data consistency under high concurrency. The lock-free design reduces CPU utilization by 60%.

[0037] The technical process for batch elimination and lookup verification in this application is as follows: a four-step interval-based batch elimination process of "atomic acquisition of Head pointer → calculation of elimination interval → batch marking of invalid indexes → atomic update of Head and Count"; and a dual key-value lookup verification mechanism of "hash index positioning → key ID comparison → timestamp verification → marking of invalid indexes".

[0038] This application adopts CAS lock-free concurrency control technology, which manages all core metadata through 64-bit atomic variables, controls the number of operation retries to within 3 times, reduces CPU utilization by 60%, and stabilizes high-concurrency response latency within 1ms; it innovates the interval-based batch elimination process, decouples elimination operations from data volume, improves elimination efficiency by 1000 times, and eliminates 1000 data entries in ≤0.2ms.

[0039] In some embodiments of this application, when Count≥Length×β, a scaling mechanism is triggered. The scaling process is executed in an independent thread, where β is a set proportional threshold, such as 0.8. The expansion process first allocates new contiguous memory, copies valid data, and then uses CAS atomic updates to update the array pointer, achieving smooth expansion with an expansion time of less than 1ms. The expansion operation is performed in a separate thread combined with CAS atomic updates to achieve non-blocking parallelism, keeping the expansion time within 1ms. A "index positioning + dual verification" lookup mechanism is designed, using Key ID comparison and timestamp verification to ensure lookup accuracy, while marking invalid indexes to avoid interference and improve operational reliability.

[0040] This application's data structure optimizes the storage structure, eliminates the predecessor / successor pointers of the doubly linked list, and adopts a compact storage mode, improving memory utilization by over 30% in small data scenarios. It also solves the problem of excessive memory consumption caused by pointer redundancy and adapts to the resource-constrained characteristics of edge devices. The insertion and deletion operation logic is simplified by integrating the multi-step synchronous operation of "hash table + linked list" into an atomic operation within a single component, reducing CPU resource consumption and ensuring that the response latency in high-concurrency scenarios (tens of thousands of reads and writes per second) remains stable within 1ms, meeting low-latency requirements.

[0041] This application's data structure implements a batch eviction mechanism, supporting "eviction of N data entries in a single operation." Eviction efficiency is decoupled from the amount of data to be evicted, keeping the eviction time for thousands of data entries within 10ms, rapidly releasing memory and handling sudden data spikes. It employs contiguous memory allocation and node reuse strategies to optimize storage layout, reducing memory fragmentation to below 10%, avoiding premature eviction due to fragmentation, and improving memory utilization efficiency.

[0042] This application also proposes a first-in-first-out (FIFO) data processing method based on key-value pair lookup, implemented based on the aforementioned FIFO data structure, including: Receive external key-value operation requests and locate the data position in the circular array through index mapping; A circular array with contiguous memory is used to manage the physical storage and order of data through compact data units and head and tail pointers; The core state parameters are stored using atomic variables, and lock-free synchronization is achieved through CAS operations to maintain the running state of the data structure.

[0043] Specifically, the core operations include data insertion, key-value lookup, FIFO eviction and batch eviction. Each operation is implemented through the collaboration of "hash index layer - ring array layer - metadata layer". The operation process is clear and the time complexity is O(1). The following is a detailed explanation with reference to the flowchart.

[0044] 3.1 Data insertion operation: Four-step atomic implementation with no component synchronization overhead. The core of the insertion operation is "write data first, then update the index," avoiding the synchronization problem of the traditional structure "dual writes to hash tables and linked lists." The process is as follows: Figure 2 As shown, the specific steps are as follows: S311: Key preprocessing — Input the raw key (such as device ID "Dev_001"), generate a 64-bit Key ID and a 32-bit hash digest using the MurmurHash3 (non-cryptographic hash function) algorithm, and calculate the initial position of the hash index layer Index_H = hash digest % hash table length.

[0045] S312: Metadata Atomic Acquisition — The current Tail pointer position (Tail_Curr) and Count value are atomically obtained through CAS operation to determine whether resizing is needed: If Count≥Length×0.8, asynchronous resizing is triggered, and the Length value is updated after the resizing is completed; if the array is full (Count==Length), a single FIFO eviction is triggered (operation 3.3 is executed), and the Head pointer is updated.

[0046] S313: Data is written to the circular array - the Key ID, variable-length Value (with length prefix), and current timestamp are written to the Tail_Curr position of the circular array. The writing process is implemented by memory copying and takes ≤0.05ms.

[0047] S314: Index Update and Metadata Synchronization - Insert the entry "Hash Digest + Tail_Curr + Valid Identifier" at the Index_H position of the hash index layer. If there is a conflict, find a free position by linear probing. After the insertion is completed, atomically update the Tail pointer (Tail_New=(Tail_Curr+1)%Length) and the Count value (Count=Count+1) through CAS operation, and the insertion operation ends.

[0048] 3.2 Key-value lookup operation: direct index access, efficient conflict handling The core of the search operation is "hash index positioning + key ID verification", avoiding traditional linked list traversal. The process is as follows: Figure 3 As shown, the specific steps are as follows: S321: Key preprocessing—Input the key to be searched, generate the Key ID and 32-bit hash digest using the same algorithm as for insertion, and calculate the initial index position Index_H.

[0049] S322: Hash Index Layer Probe - Starting from the Index_H position, traverse the hash index entries, matching "hash digest + valid identifier". If a matching entry is found, obtain the corresponding circular array index Index_A; if the traversal reaches the end of the collision chain and still does not find it, return "search failed".

[0050] S323: Data validity verification - Access the Index_A position of the circular array, read the stored Key ID_Store and timestamp, and determine whether the Key ID is consistent with the Key ID to be searched and the timestamp has not expired (the expiration time can be configured, and the default is no expiration).

[0051] S324: Result return - If the verification passes, read the variable-length Value field and parse and return it; if the verification fails (Key ID does not match or has expired), mark the hash index entry as "invalid", return "search failed", and the search operation ends.

[0052] 3.3 FIFO Eviction Operation: Pointer Movement, Batch Index Cleaning To address the problem of low single-node eviction efficiency in the existing structure, this application supports two modes: single eviction and batch eviction. The core is "batch pointer movement + index range cleaning", and the process is as Figure 4 shown. The specific steps are as follows: S331: Eviction Trigger - When the array is full (Count == Length) or the memory usage rate ≥ 95%, trigger the eviction mechanism, and determine the eviction quantity N according to the requirements (N = 1 for single eviction, and N for batch eviction is calculated based on the memory release requirement).

[0053] S332: Eviction Range Calculation - Atomically obtain the current Head pointer position Head_Curr through CAS, calculate the eviction end position Head_New = (Head_Curr + N) % Length, and determine the eviction range as [Head_Curr, Head_New) (if Head_Curr < Head_New, it is a continuous range, otherwise it is divided into two parts: [Head_Curr, Length) and [0, Head_New)).

[0054] S-333: Batch Hash Index Cleaning - Traverse all data units within the eviction range of the circular array, read the KeyID and calculate the hash digest, locate the corresponding entry in the hash index layer, and mark it as "invalid" to achieve batch cleaning. The cleaning efficiency is 1000 times higher than single-node eviction. [[ID=二十]]

[0055] S334: Metadata Update - Atomically update the Head pointer to the Head_New position through a CAS operation, and at the same time reduce the Count value by N (Count = Count - N). The eviction operation ends, and the memory space is released.

[0056] The "hash-ring array" collaborative architecture of this application eliminates pointer redundancy in doubly linked lists, improving memory utilization by more than 30%; secondly, the lock-free concurrency control mechanism reduces CPU overhead, and the response latency is stable within 1ms under high concurrency; thirdly, the interval batch eviction mechanism decouples eviction efficiency from data volume, and the time to eviction 1000 data items is ≤0.2ms.

[0057] Test data in edge gateway scenarios (8MB memory, 800MHz CPU) show that, compared with JavaLinkedHashMap, this solution reduces the time for single insertion from 0.5ms to 0.1ms, the time for single lookup from 0.3ms to 0.08ms, and the amount of data that can be stored in 8MB memory from 180,000 to 330,000 records, fully adapting to the resource constraints and performance requirements of edge computing and IoT.

[0058] It should be noted that, in the embodiments of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0059] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0060] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0061] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims. All of these forms are within the protection scope of this application.

Claims

1. A first-in-first-out (FIFO) data structure based on key-value pair lookup, characterized in that, The architecture comprises a three-layer closed-loop system: hash index, circular array, and metadata control. Each layer collaborates through a closed-loop system of index mapping, data storage, and state management. The hash index layer is used to receive external key-value operation requests and locate the data position in the circular array through index mapping. The hash index layer adopts a storage design of hash digest and index. The circular array storage layer is used for physical storage and sequential management of data. It adopts a circular array with contiguous memory and achieves FIFO sequential management through compact data units and head and tail pointers. The metadata control layer maintains the structure's operating state and provides parameter support for operation execution. The metadata control layer uses atomic variables to store core state parameters and achieves lock-free synchronization through CAS operations.

2. The first-in-first-out data structure based on key-value pair lookup as described in claim 1, characterized in that, The hash index layer includes a hash digest of a specified number of bits, a circular array index, and a status identifier for each index entry, with a total length not exceeding 64 bits. The status identifier, which is used to mark the validity of the entry, is 8 bits long.

3. The first-in-first-out data structure based on key-value pair lookup as described in claim 2, characterized in that, The hash index layer employs a hybrid approach of linear probing and collision chaining. In the event of a collision at the initial index position, it probes for free positions in the order of index + 1, and records the collision chain through a status identifier.

4. The first-in-first-out data structure based on key-value pair lookup as described in claim 1, characterized in that, The structural parameters of the circular array in the circular array storage layer are: The initial length is 2^n. The array uses contiguous memory allocation. Each expansion is doubled in length. During expansion, valid data is migrated in batches using a memory copy function. Each of the compact data units in the circular array storage layer includes: Key ID, which is consistent with the length of the hash index layer; The Value field uses a length prefix encoding, which includes a 1-byte length identifier and the data content; Timestamps are used to help verify the validity of data.

5. The first-in-first-out data structure based on key-value pair lookup as described in claim 4, characterized in that, The head and tail pointers of the circular array storage layer include a head pointer and a tail pointer, wherein, The head pointer points to the earliest inserted data position; The tail pointer points to the position to be inserted. If the array is not full, the tail pointer moves by Tail=(Tail+1)%Length, where Length is the length of the circular array; When the array is full, new data overwrites the head pointer position, and the head pointer moves synchronously.

6. The first-in-first-out data structure based on key-value pair lookup as described in claim 5, characterized in that, The metadata control layer contains core metadata, including the length of the circular array, the current data volume, the position of the head pointer, the position of the tail pointer, and the load factor of the hash index layer. All core metadata is stored with the same length as the hash index layer.

7. The first-in-first-out data structure based on key-value pair lookup as described in claim 6, characterized in that, In the case of insertion / deletion operations, the Head / Tail pointers and Count value are atomically updated using CAS operations.

8. The first-in-first-out data structure based on key-value pair lookup as described in claim 6, characterized in that, When Count ≥ Length × β, the expansion mechanism is triggered. The expansion process is executed in a separate thread, where β is a set proportional threshold. To expand the memory, first allocate new contiguous memory, copy the valid data, and then update the array pointer using CAS atomic operations.

9. A first-in-first-out (FIFO) data processing method based on key-value pair lookup, characterized in that, Based on the first-in-first-out data structure as described in any one of claims 1-8, including: Receive external key-value operation requests and locate the data position in the circular array through index mapping; A circular array with contiguous memory is used to manage the physical storage and order of data through compact data units and head and tail pointers; The core state parameters are stored using atomic variables, and lock-free synchronization is achieved through CAS operations to maintain the running state of the data structure.

Images