Sequence data hierarchical storage method and system based on data processing unit

Abstract

The invention discloses a sequence data hierarchical storage method and system based on a data processing unit, and the method comprises the steps: dividing sequence data into stock data stored in a persistent storage device and incremental data cached in a host memory; the DPU performs high-speed matching and identification on request types, and classifies the request types into additional requests and query requests. And for the additional request, the DPU executes a fast path forwarding strategy and directly routes the data to an incremental data area of a host memory, so that low-delay writing is realized. For the query request, the DPU executes data preprocessing of hardware acceleration, carries out push filtering, mode matching or decompression on stock data, and only transmits filtered effective result data to a host CPU for subsequent processing; intelligent shunting and preprocessing of read-write requests are realized by utilizing the DPU at a system entrance, and a physically isolated read-write path is constructed, so that the overhead of a host CPU (Central Processing Unit) is remarkably reduced, and the query performance and the write-in throughput of large-scale sequence data are effectively improved.

Description

Technical Field

[0001] This invention relates to the field of data storage technology, and in particular to a hierarchical storage method and system for sequential data based on a data processing unit. Background Technology

[0002] In related technologies, with the rapid development of fintech, the Internet of Things (IoT), and internet applications, sequential data has become a core data asset. This type of data is continuously and incrementally written sequentially (e.g., chronologically), exhibiting strong temporal locality: the access frequency of the data is highly concentrated within the most recently generated time window, naturally forming historically stable "stock data" and frequently changing "incremental data." To efficiently manage this type of data, existing technologies commonly employ hierarchical storage or architectures similar to log-merged tree (LSM-Tree). This architecture typically stores the latest "incremental data" in high-speed memory or cache to meet high-throughput write and real-time query requirements; while historical "stock data" is compressed and archived into lower-cost persistent storage media (such as SSDs or HDDs) to reduce storage costs.

[0003] However, in traditional host CPU-centric computing architectures, this tiered storage model exposes severe bottlenecks when facing high-concurrency requests. This architecture delegates all data plane tasks, including network protocol stack processing, system interrupts, data copying, and even storage driver interactions, to the host CPU. High-throughput write requests trigger frequent context switching, continuously consuming CPU computing power, while queries on historical "existing data" are even more demanding. This not only involves massive data movement from persistent storage to memory, consuming valuable PCIe bus bandwidth, but more importantly, the host CPU must first decompress compressed data blocks before performing filtering, resulting in a significant waste of computing power on data that will ultimately be discarded, causing serious data movement overhead and wasted computing resources. This design, placing two workloads with vastly different characteristics—reading (querying) and writing (writing)—on the same physical execution path leads to severe resource contention. High-intensity write flows crowd out the bus and memory bandwidth required for queries, and conversely, computationally intensive query tasks can block the write path, causing mutual interference and unpredictability in performance. This tightly coupled architecture makes system optimization extremely difficult, making it hard to simultaneously meet the business requirements of high-throughput writes and low-latency queries. Summary of the Invention

[0004] This invention aims to at least partially solve one of the technical problems in the aforementioned technologies. To this end, one objective of this invention is to propose a hierarchical storage method for sequential data based on a data processing unit. By deploying a data processing unit at the system front end, a host-independent, programmable data plane is constructed, enabling intelligent identification, routing, and hardware acceleration of read and write requests, thereby improving resource utilization, reducing query latency, and ensuring service quality.

[0005] To achieve the above objectives, this invention proposes a hierarchical storage method for sequential data based on a data processing unit, applied to a hierarchical storage system for sequential data based on a data processing unit. The system includes a host, a persistent storage device, and a data processing unit. The host includes a central processing unit and host memory. The host memory is divided into an incremental data area for storing incremental data, and the persistent storage device stores compressed existing data. The method includes the following steps: using the data processing unit to obtain external data requests, and identifying the request type of the external data request according to preset rules; if the request type is an append request, then writing the data content in the external data request into the incremental data area of ​​the host memory; if the request type is a query request, then reading data according to the query range in the external data request, wherein when the query range involves existing data, reading the corresponding compressed existing data block from the persistent storage device, and decompressing and filtering the existing data block to obtain the existing data.

[0006] According to the hierarchical storage method for sequential data based on a data processing unit (DPU) according to embodiments of the present invention, a host-independent, programmable data plane is constructed to completely offload the I / O load of infrastructure such as network and storage from the host's central processing unit (CPU). The DPU, as the hardware carrier of this data plane, is capable of performing line-rate online parsing, preprocessing, and distribution of the data stream along the data path. In this architecture, the CPU is freed up to function as a control plane, focusing on upper-layer business logic. This clear separation of the data plane and control plane provides strong architectural support for resolving I / O bottlenecks and resource contention issues in sequential data processing.

[0007] In addition, the sequence data hierarchical storage method based on data processing units proposed in the above embodiments of the present invention may also have the following additional technical features:

[0008] Optionally, the request type of the external data request is identified according to a preset rule, including: using the hardware regular expression parsing accelerator of the data processing unit to identify the request type of the external data request according to the preset rule, wherein the preset rule is a regular expression rule, and the external data request is classified in real time by matching the request content with the regular expression rule online.

[0009] Optionally, before the data processing unit obtains an external data request, the method further includes: constructing and initializing a circular buffer within the incremental data area, and initializing an atomic pointer for managing the circular buffer in the shared memory of the data processing unit, so that the data processing unit can safely reserve write space in the circular buffer for the append request according to the atomic pointer.

[0010] Optionally, writing the data content in the external data request into the incremental data area of ​​the host includes: using remote direct memory access technology to write the data content into the incremental data area.

[0011] Optionally, before the data processing unit obtains external data requests, the method further includes: constructing and maintaining a master index structure covering the existing data in the central processing unit of the host, and synchronizing a copy of the master index structure to the data processing unit so that the data processing unit can perform local lookup when performing existing data reading.

[0012] Optionally, reading the corresponding compressed existing data block from the persistent storage device and decompressing and filtering the existing data block includes: the host's central processing unit parsing the query request to generate a query task including the location information of the data block to be read and filtering conditions, and sending it to the data processing unit through a communication queue; the data processing unit receiving the query task and directly reading the corresponding compressed existing data block from the persistent storage device into the onboard memory of the data processing unit according to the location information; and the hardware regular expression parsing accelerator of the data processing unit decompressing the data stream in the onboard memory and performing online filtering according to the filtering conditions.

[0013] Optionally, when the query scope involves incremental data, the incremental data index is queried in the host memory to obtain the corresponding incremental data.

[0014] Optionally, when the query scope involves both incremental and existing data, after obtaining the incremental and existing data, the method further includes: merging the incremental and existing data to obtain the final read data, and sending the final read data to the requester.

[0015] Optionally, the host, persistent storage device, and data processing unit communicate via a PCIe bus.

[0016] To achieve the above objectives, a first aspect of the present invention proposes a hierarchical storage system for sequential data based on a data processing unit, comprising: a host, a persistent storage device, and a data processing unit. The host includes a central processing unit and host memory. The host memory is divided into an incremental data area for storing incremental data, and the persistent storage device stores compressed existing data.

[0017] The data processing unit is used to acquire external data requests and identify the request type of the external data requests according to preset rules. If the request type is an append request, the data content in the external data request is written into the incremental data area of ​​the host memory. If the request type is a query request, data is read according to the query range in the external data request. When the query range involves existing data, the corresponding compressed existing data block is read from the persistent storage device, and the existing data block is decompressed and filtered to obtain existing data that meets the query conditions. Attached Figure Description

[0018] Figure 1 This is a flowchart illustrating a hierarchical storage method for sequence data based on a data processing unit according to an embodiment of the present invention.

[0019] Figure 2 This is a schematic diagram of the hardware structure of a sequence data hierarchical storage system based on a data processing unit according to an embodiment of the present invention;

[0020] Figure 3 This is a flowchart illustrating the system initialization and preparation phase according to an embodiment of the present invention.

[0021] Figure 4 This is a flowchart illustrating the key data structures and logical layout according to an embodiment of the present invention;

[0022] Figure 5 This is a schematic diagram of the data flow for processing append requests according to an embodiment of the present invention;

[0023] Figure 6 This is a schematic diagram of the data flow for processing query requests according to an embodiment of the present invention. Detailed Implementation

[0024] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0025] To better understand the above technical solutions, exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the present invention and to fully convey the scope of the invention to those skilled in the art.

[0026] To better understand the above technical solutions, the following will provide a detailed explanation of the technical solutions in conjunction with the accompanying drawings and specific implementation methods.

[0027] Reference Figure 2 The illustrated hierarchical storage system for sequential data based on a data processing unit includes a host, persistent storage devices, and a data processing unit. The host includes a central processing unit and host memory. The host memory is divided into incremental data areas for storing incremental data, and the persistent storage devices store compressed existing data. Specifically:

[0028] Host: A standard server node equipped with a multi-core central processing unit (CPU) and main memory; the software running on the host serves as the system's control plane, responsible for running the operating system, upper-layer business applications, managing data indexes, parsing complex queries, and aggregating the final results.

[0029] Persistent storage device (disk): It is preferable to use one or more high-performance NVMe SSDs to persistently store compressed historical data, forming a "cold data layer" of data.

[0030] Data Processing Unit (DPU): The DPU connects to the host via the PCIe high-speed bus and serves as the system's data plane, responsible for handling all I / O traffic entering and leaving the server. The DPU integrates a request parsing module, a task processing module, and a data processing acceleration module. The request parsing module efficiently runs the network protocol stack and the data forwarding and preprocessing logic in this invention. The task processing module receives tasks from the host and maintains necessary data information in memory. The data processing acceleration module efficiently moves and processes data within the DPU, between the DPU and host memory, and between the DPU and persistent storage.

[0031] Figure 1This is a flowchart illustrating a hierarchical storage method for sequential data based on a data processing unit according to an embodiment of the present invention, applied to the aforementioned hierarchical storage system for sequential data based on a data processing unit, such as... Figure 1 As shown, the method includes the following steps:

[0032] S101, the data processing unit acquires external data requests and identifies the request type of the external data requests according to preset rules.

[0033] As an example, identifying the request type of external data requests according to preset rules includes: using a hardware regular expression parsing accelerator of the data processing unit to identify the request type of external data requests according to preset rules, wherein the preset rules are regular expression rules, and real-time classification of external data requests is achieved by matching the request content with the regular expression rules online.

[0034] It should be noted that HTTP POST requests matching the URL / append are identified as append requests, and GET requests matching / query are identified as query requests.

[0035] S102, if the request type is an append request, then write the data content in the external data request to the incremental data area of ​​the host memory.

[0036] As one embodiment, before the data processing unit obtains an external data request, the method further includes: constructing and initializing a circular buffer in the incremental data area, and initializing an atomic pointer for managing the circular buffer in the shared memory of the data processing unit, so that the data processing unit can safely reserve write space in the circular buffer for append requests based on the atomic pointer.

[0037] As one embodiment, writing the data content in an external data request to the incremental data area of ​​the host includes: using Remote Direct Memory Access (RDMA) technology to write the data content to the incremental data area.

[0038] S103, if the request type is a query request, then the data is read according to the query range in the external data request. When the query range involves existing data, the corresponding compressed existing data block is read from the persistent storage device, and the existing data block is decompressed and filtered to obtain the existing data.

[0039] As an example, before the data processing unit obtains external data requests, the method further includes: building and maintaining a main index structure covering the existing data in the host's central processing unit, and synchronizing a copy of the main index structure to the data processing unit so that the data processing unit can perform local lookup when performing existing data reading. Since the incremental data absorbs real-time data writing, the index main index structure changes less, and the scope of impact can be reduced through data sharding, so the frequency of index synchronization is low.

[0040] As one embodiment, reading the corresponding compressed existing data block from the persistent storage device and decompressing and filtering the existing data block includes: the host's central processing unit parsing the query request to generate a query task including the location information of the data block to be read and the filtering conditions, and sending it to the data processing unit through a communication queue; the data processing unit receiving the query task and reading the corresponding compressed existing data block directly from the persistent storage device into the onboard memory of the data processing unit according to the location information; and the hardware regular expression parsing accelerator of the data processing unit decompressing the data stream in the onboard memory and performing online filtering according to the filtering conditions.

[0041] To better understand the above technical solution, a detailed description is provided through the following specific embodiment, with reference to... Figure 3 During system startup or business application initialization, the following preparatory work needs to be performed to establish an efficient and low-latency collaboration mechanism between the host CPU (control plane) and DPU (data plane):

[0042] (1) Memory allocation and registration:

[0043] The control plane software on the host requests a large block of main memory from the operating system as an incremental data area to store recently accessed data. Subsequently, the application, through the DPU driver, registers the physical address, size, and other metadata of this incremental data area with the DPU. This step is crucial, as it grants the DPU hardware direct read-write (RDMA) access to this host memory area, laying the foundation for subsequent zero-copy writes.

[0044] (2) Initialization of indexes and collaborative data structures:

[0045] Appendix Write Communication Management: To achieve efficient data writing from the DPU to the host, the host CPU constructs and initializes a ring buffer within the registered incremental data area. Simultaneously, in a shared memory area accessible to the DPU, atomic pointers for managing this buffer are initialized. These primarily include: a DPU write position pointer (pointing to the next writable buffer position for the DPU), a committed data position pointer (pointing to the last valid data position that has been written), and a consumed data position pointer (pointing to the data position where the host has completed indexing the incremental data). These atomic pointers ensure concurrent access between the DPU and the host without lock operations, implementing an efficient producer-consumer model.

[0046] Existing data management: The host CPU builds and maintains a primary index structure (e.g., a B+ tree) covering all historical data in host memory. This primary index is responsible for mapping high-level query conditions (such as timestamp ranges, tags, etc.) to the physical addresses of corresponding existing data blocks on persistent storage. Simultaneously, a copy of this primary index is synchronized to the DPU's storage (memory and persistent storage). The DPU can cache frequently accessed indexes in memory for fast local lookups during query unloading.

[0047] Lightweight communication queue establishment: A lightweight message queue is established between the host and the DPU for metadata exchange and "task" distribution. The host sends control commands such as query tasks to the DPU.

[0048] DPU rule programming: The host uses the interface provided by the DPU to program and load the regular expression rules used to distinguish request types (for example, identifying an HTTP POST request matching the URL / append as an append request, and a GET request matching / query as a query request) into the hardware regular expression computing accelerator of the DPU, enabling it to intelligently identify and distribute network traffic at line speed.

[0049] Reference Figure 4 and Figure 5 As shown, the append request processing flow includes:

[0050] (1) An append request data packet sent by an external client reaches the DPU via the network.

[0051] (2) The data packet is guided to the regularization accelerator by the hardware pipeline of the DPU and is successfully identified as an "append request" according to the preset rules.

[0052] (3) The processing unit on the DPU performs an atomic "compare-and-swap" operation on the DPU write position pointer in the shared memory. This operation can safely and lock-free reserve write space in the incremental data circular buffer for the request, ensure that concurrent writes are conflict-free, and return the exact physical address of this write.

[0053] (4) The DPU's DMA engine is triggered, performing an RDMA write operation. It writes the valid data payload from the request packet directly from the DPU's internal cache, across the PCIe bus, to the host memory physical address obtained in the previous step. Throughout the entire data write process, the host CPU is completely uninvolved, with no interruptions or data copying, achieving "zero-copy" writing. After the data write is complete, the DPU updates the committed data position pointer in strict order. This update operation makes the newly written data officially visible to the host CPU. The host application can asynchronously poll the difference between the committed data position pointer and the consumed data position pointer to process newly arrived data in batches.

[0054] (5) After consuming incremental data, the host CPU will update the incremental data index in memory according to its content. When the incremental data meets a specific policy (such as reaching a certain size or exceeding the time window), the host will trigger an asynchronous flush process to compress this part of the data and write it to persistent storage, making it part of the existing data, and update the existing data master index on the host and DPU accordingly.

[0055] Reference Figure 4 and Figure 6 As shown, the query request processing flow includes:

[0056] (1) A query request data packet sent by an external client arrives at the DPU and is also recognized as a "query request" by the regular expression computing accelerator.

[0057] (2) The DPU forwards the lightweight query request metadata directly to the host. The host, as the control plane, is responsible for the comprehensive parsing and planning of the query request.

[0058] (2-1) Scenario 1: The query range only covers incremental data in memory. The host discovers that the requested data is entirely within the incremental data area by using the query range (e.g., time period) of the requested key. In this case, the host will directly complete the query, calculation, and return the result in memory, resulting in extremely low latency.

[0059] (2-2) Scenario Two: The query scope involves existing data in persistent storage. This is the core path of hardware acceleration: a. After the host CPU parses the query conditions for the existing data, it packages one or more queries into a "processing task" and sends it to the DPU through a communication queue. b. After receiving the task, the application on the DPU parses the list of physical addresses of the data blocks to be read and the filtering conditions to be executed based on the query task and the index. It uses its storage offloading capability to directly initiate read commands to the disk, accurately reading one or more compressed data blocks into the DPU's own onboard memory. This process can bypass the host memory. c. The read data stream is directly sent to the DPU's hardware decompression accelerator for decompression. The decompressed data stream is then sent to the hardware filtering accelerator (which can reuse the regular expression parsing accelerator or dedicated logic). The accelerator performs online filtering based on the query conditions specified in the host task. d. Only valid result data that meets the conditions and is much smaller than the original data blocks is retained and sent to the host in batches at once via the PCIe bus.

[0060] (3) Result aggregation: After receiving the preprocessing results of the existing data from the DPU, the host CPU combines them with its own query results of the incremental data in memory to perform the final data merging, sorting or aggregation, forming a complete query response and sending it to the client.

[0061] Through the above embodiments, this application utilizes the DPU to construct a high-efficiency, independent data plane, completely freeing the host CPU from heavy low-level I / O and data preprocessing tasks, allowing it to focus on high-level control and business logic. By separating read and write paths at the hardware level and accelerating query pushdown, resource utilization is greatly optimized, resolving the performance contention problem in traditional architectures, thereby achieving high-performance, low-latency sequential data processing.

[0062] To implement the above embodiments, this invention also proposes a hierarchical storage system for sequential data based on a data processing unit, such as... Figure 2 As shown, the sequence data hierarchical storage system based on the data processing unit includes: a host 10, a persistent storage device 20, and a data processing unit 30.

[0063] The host 10 includes a central processing unit and host memory. The host memory is divided into an incremental data area for storing incremental data, and the persistent storage device stores compressed existing data.

[0064] The data processing unit 30 is used to acquire external data requests and identify the request type of the external data request according to preset rules. If the request type is an append request, the data content in the external data request is written into the incremental data area of ​​the host memory. If the request type is a query request, the data is read according to the query range in the external data request. When the query range involves existing data, the corresponding compressed existing data block is read from the persistent storage device 20, and the existing data block is decompressed and filtered to obtain existing data that meets the query conditions.

[0065] In summary, the hierarchical storage method and system for sequence data based on data processing units of this application have the following beneficial effects:

[0066] 1. Significantly reduces host CPU load and improves resource utilization.

[0067] By deploying a DPU at the system front end to build an independent data plane, the heavy-duty low-level infrastructure tasks originally handled by the host CPU, such as network protocol processing, I / O request classification, data filtering, pattern matching, and decompression, are completely offloaded to the DPU's dedicated hardware acceleration engine. This frees the host CPU from the specific execution of the data plane, allowing it to focus more on upper-layer business logic as the control plane, thereby greatly reducing its resource consumption and increasing the effective computing power of a single server and application deployment density.

[0068] 2. Significantly improve query performance and efficiency, and reduce query latency.

[0069] Using a DPU for decompression and filtering is far more efficient than a CPU-based software implementation. The combination of these two methods reduces latency for complex queries on historical data.

[0070] 3. Achieve true hardware read / write separation to ensure Quality of Service (QoS).

[0071] For high-throughput "add" requests and computationally intensive "query" requests, physically isolated processing paths are planned on the DPU: the former directly accesses host memory via technologies such as RDMA, while the latter enters the hardware-accelerated processing pipeline. This hardware-level separation at the system entry point completely avoids contention for host CPU, memory bus, and other resources between the two different loads of reading and writing, eliminating interference and ensuring that the system can still provide stable and predictable performance in high-concurrency mixed read and write scenarios.

[0072] 4. Improve the overall scalability and flexibility of the system architecture.

[0073] This architecture decouples data processing and I / O capabilities from the host CPU's capabilities. In the future, when there is a need to increase the system's network or data processing throughput, this can be achieved by independently upgrading to a higher-specification DPU, without requiring costly upgrades or replacements of the host and the applications running on it. This architecture provides a more cost-effective and flexible path for system expansion.

[0074] It should be noted that the above description and examples of the sequence data hierarchical storage method based on data processing units are also applicable to the sequence data hierarchical storage system based on data processing units in this embodiment, and will not be repeated here.

[0075] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0076] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0077] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0078] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0079] It should be noted that any reference signs placed between parentheses in the claims should not be construed as limiting the claims. The word "comprising" does not exclude the presence of components or steps not listed in the claims. The word "a" or "an" preceding a component does not exclude the presence of a plurality of such components. The invention can be implemented by means of hardware comprising several different components and by means of a suitably programmed computer. In a unit claim enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names.

[0080] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0081] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

[0082] In the description of this invention, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0083] In this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," "linking," and "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0084] In this invention, unless otherwise explicitly specified and limited, "above" or "below" the second feature can mean that the first feature is in direct contact with the second feature, or that the first feature is in indirect contact with the second feature through an intermediate medium. Furthermore, "above," "over," and "on top" of the second feature can mean that the first feature is directly above or diagonally above the second feature, or simply that the first feature is at a higher horizontal level than the second feature. "Below," "below," and "under" the second feature can mean that the first feature is directly below or diagonally below the second feature, or simply that the first feature is at a lower horizontal level than the second feature.

[0085] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms should not be construed as necessarily referring to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0086] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A hierarchical storage method for sequence data based on a data processing unit, characterized in that, A hierarchical storage system for sequential data based on a data processing unit is provided. The system includes a host, a persistent storage device, and a data processing unit. The host includes a central processing unit and host memory. The host memory is divided into an incremental data area for storing incremental data. The persistent storage device stores compressed existing data. The method includes the following steps: An external data request is acquired using a data processing unit, and the request type of the external data request is identified according to a preset rule. If the request type is an append request, the data content in the external data request is written to the incremental data area of ​​the host memory; If the request type is a query request, data is read according to the query range in the external data request. When the query range involves existing data, the corresponding compressed existing data block is read from the persistent storage device, and the existing data block is decompressed and filtered to obtain the existing data.

2. The hierarchical storage method for sequence data based on a data processing unit as described in claim 1, characterized in that, Identifying the request type of the external data request according to preset rules includes: using the hardware regular expression parsing accelerator of the data processing unit to identify the request type of the external data request according to preset rules, wherein the preset rules are regular expression rules, and real-time classification of the external data request is achieved by online matching of the request content with the regular expression rules.

3. The hierarchical storage method for sequence data based on a data processing unit as described in claim 1, characterized in that, Before the data processing unit obtains an external data request, the method further includes: constructing and initializing a circular buffer in the incremental data area, and initializing an atomic pointer for managing the circular buffer in the shared memory of the data processing unit, so that the data processing unit can safely reserve write space in the circular buffer for the append request according to the atomic pointer.

4. The hierarchical storage method for sequence data based on a data processing unit as described in claim 3, characterized in that, Writing the data content from the external data request into the incremental data area of ​​the host includes: using remote direct memory access technology to write the data content into the incremental data area.

5. The hierarchical storage method for sequence data based on a data processing unit as described in claim 1, characterized in that, Before the data processing unit obtains external data requests, the method further includes: constructing and maintaining a master index structure covering the existing data in the central processing unit of the host, and synchronizing a copy of the master index structure to the data processing unit so that the data processing unit can perform local lookup when performing existing data reading.

6. The hierarchical storage method for sequence data based on a data processing unit as described in claim 5, characterized in that, Reading the corresponding compressed existing data block from the persistent storage device, and decompressing and filtering the existing data block, including: The central processing unit of the host parses the query request to generate a query task including the location information of the data block to be read and the filtering conditions, and sends it to the data processing unit through the communication queue; The data processing unit receives the query task and reads the corresponding compressed stored data block directly from the persistent storage device into the onboard memory of the data processing unit according to the location information. The hardware regular expression parsing accelerator of the data processing unit decompresses the data stream in the onboard memory and performs online filtering according to the filtering conditions.

7. The hierarchical storage method for sequence data based on a data processing unit as described in claim 6, characterized in that, When the query scope involves incremental data, the incremental data index is queried in the host memory to obtain the corresponding incremental data.

8. The hierarchical storage method for sequence data based on a data processing unit as described in claim 7, characterized in that, When the query scope involves both incremental and existing data, after obtaining the incremental and existing data, it also includes: The incremental data and existing data are merged to obtain the final read data, and the final read data is sent to the requester.

9. The hierarchical storage method for sequence data based on a data processing unit as described in any one of claims 1-7, characterized in that, The host, persistent storage device, and data processing unit communicate via the PCIe bus.

10. A hierarchical storage system for sequential data based on a data processing unit, characterized in that, include: The system includes a host, a persistent storage device, and a data processing unit. The host includes a central processing unit and host memory. The host memory is divided into an incremental data area for storing incremental data. The persistent storage device stores compressed existing data. The data processing unit is used to acquire external data requests and identify the request type of the external data requests according to preset rules. If the request type is an append request, the data content in the external data request is written into the incremental data area of ​​the host memory. If the request type is a query request, data is read according to the query range in the external data request. When the query range involves existing data, the corresponding compressed existing data block is read from the persistent storage device, and the existing data block is decompressed and filtered to obtain existing data that meets the query conditions.

Images