Embedded storage device and method for high-speed streaming data
By designing an embedded storage device and method for high-speed streaming data, and employing hardware and software memory address mapping and scatter-gather DMA technology, the poor hard disk I/O performance and compatibility issues in existing storage solutions are resolved, achieving efficient streaming data storage and transmission.
Patent Information
- Application Number
- CN202511048874.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-07-29
- Publication Date
- 2025-11-11
AI Technical Summary
Existing storage solutions suffer from poor hard disk I/O performance, high software overhead, and poor compatibility in high-speed streaming data scenarios, failing to effectively reduce unnecessary processor load and improve storage efficiency.
An embedded storage device and method for high-speed streaming data is adopted. A custom block device driver is designed through hardware and software memory address mapping and address replacement. Combined with scatter-gather DMA technology, the transfer of streaming data in processor memory is reduced and the data transfer path is optimized.
It improves the efficiency of streaming data storage, reduces processor resource utilization, enhances data transmission efficiency, and supports compatibility with different hardware platforms.
Smart Images
Figure CN120929016A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of big data technology, specifically relating to an embedded storage device and method for high-speed streaming data. Background Technology
[0002] In the field of big data, the need for storing streaming data is widespread. For example, in industrial control systems, the data generated by AD acquisition cards (analog-to-digital conversion acquisition cards) is typically high-frequency, high-precision streaming data, which needs to be stored in real time for subsequent analysis and control decisions. Furthermore, in high-speed network environments (such as 5G and data centers), massive amounts of streaming data are generated. To accomplish real-time data processing and analysis, streaming data storage faces stringent requirements for throughput, low latency, and data integrity.
[0003] In the scenarios requiring efficient streaming data storage, existing storage solutions are not perfect. First, under high-speed, frequent disk I / O, existing solutions incur significant software overhead, including interrupts, context switching, and memory copying, which greatly increases the processor load and leads to poor disk read / write performance. Second, different disks, controllers, and host systems have varying implementations; existing solutions sacrifice some performance to ensure platform compatibility. Further research is needed to improve streaming data storage performance by minimizing unnecessary software overhead and providing maximum support for hardware platforms. Summary of the Invention
[0004] This invention provides an embedded storage device and method for high-speed streaming data to solve the aforementioned technical problems, specifically adopting the following technical solution:
[0005] An embedded storage device for high-speed streaming data includes:
[0006] Peripheral devices are used to implement streaming data transmission and storage functions;
[0007] The storage management module, in the form of a block device driver, controls the read and write operations between the peripheral device cache and the storage medium;
[0008] The hardware and software interaction module is used to interact with the peripheral device and instruct the peripheral device to complete the streaming data transmission and processing;
[0009] The file management module is used to store a mapping table between streaming data channels and stored files, thus associating streaming data channels with files.
[0010] The buffer management module is responsible for managing peripheral device caches and implementing the mapping from processor memory to peripheral device caches.
[0011] Furthermore, the storage function of the peripheral device supports bidirectional transmission of streaming data between the disk and the data buffer;
[0012] The peripheral device's transmission function supports bidirectional transmission of streaming data between the external device and the data buffer.
[0013] Furthermore, the storage management module includes:
[0014] The address mapping module stores the address mapping table between the processor memory and the peripheral device memory. It manages the address buffer through a custom linked list for the scatter-gather DMA controller to parse and process.
[0015] The request processing module is used to process I / O requests issued by the file system and generate command packages.
[0016] Furthermore, the data interaction between the software and hardware interaction module and the peripheral device is implemented based on command queues and response queues, and the software and hardware communicate through doorbell registers and interrupts.
[0017] Furthermore, the command queue and response queue are managed through a FIFO queue. The command queue is filled by software and read by peripheral devices, while the response queue is filled by peripheral devices and read by software.
[0018] Furthermore, the file management module is responsible for scheduling I / O requests and controlling the storage direction, capacity, and disk access location of streaming data;
[0019] The file management module monitors the data storage status of the streaming data in real time and returns corresponding errors to the user when errors occur.
[0020] Furthermore, the buffer management module designs an address mapping table from processor memory to peripheral device cache using a page mapping method;
[0021] The buffer management module allocates a large block of address space and obtains the corresponding processor memory physical address. It fills the physical address of the selected peripheral device cache into the corresponding address mapping table and sends it to the storage management module.
[0022] An embedded storage method for high-speed streaming data, applied to the aforementioned embedded storage device for high-speed streaming data, the embedded storage method for high-speed streaming data comprising:
[0023] S100: The buffer management module allocates the processor memory physical address and the peripheral device cache physical address, generates an address mapping table, and hands it over to the address mapping module in the storage management module;
[0024] S200: The file management module generates a write I / O request, which is processed by the file system and then passed to the storage management module;
[0025] S300: The request processing module receives I / O requests from the file system, processes them, generates corresponding command packets, and fills them into the command queue for execution.
[0026] S400: The software and hardware interaction module submits work requests to the peripheral device and notifies the peripheral device of the number of work requests sent through the doorbell register;
[0027] S500: Peripheral devices acquire and parse command packets, and move streaming data from the buffer to the disk;
[0028] S600: After the transfer is completed, the peripheral device generates a response packet and fills it into the response queue, and returns a completion signal to the software and hardware interaction module;
[0029] S700: The hardware and software interaction module receives a completion signal, reads and parses the response packet returned by the peripheral device;
[0030] S800: The file management module monitors the file writing status and stops writing data to the file when the file writing fails or the file is full.
[0031] Furthermore, in step S300, the address mapping module parses the I / O request to obtain the physical address of the I / O request buffer, and determines the actual physical address of the I / O request by combining it with the address mapping table. If the physical address of the buffer is in the address mapping table, it is replaced with the physical address of the corresponding peripheral device memory; otherwise, no address replacement is performed.
[0032] The advantage of this invention lies in the fact that the provided embedded storage device and method for high-speed streaming data adopts a software and hardware memory address mapping plus address replacement scheme to design a custom block device driver. Streaming data does not need to pass through processor memory, reducing the number of memory copies, reducing processor resource utilization, and thus improving the efficiency of streaming data storage.
[0033] The advantages of this invention also lie in the embedded storage device and method for high-speed streaming data, which, combined with scatter-gather DMA technology, effectively manages data blocks in scattered and aggregated storage, enabling the DMA controller to transfer multiple physically scattered data blocks at once, thereby improving the efficiency of streaming data transmission. Attached Figure Description
[0034] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0035] Figure 1 This is a schematic diagram of an embedded storage device for high-speed streaming data according to this application;
[0036] Figure 2 This is a schematic diagram of the command and response packet structure of the software and hardware interaction module in this application;
[0037] Figure 3 This is a schematic diagram of the embedded storage method for high-speed streaming data in this application;
[0038] Figure 4 This is a schematic diagram of the request processing module of the embedded storage method for high-speed streaming data in this application. Detailed Implementation
[0039] The embodiments of this application are described in detail below. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.
[0040] Obviously, the embodiments described in this application are merely some, not all, of the embodiments. It should be noted that those skilled in the art can combine the embodiments and features described therein without departing from the concept of this application and without conflicting with each other. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort should fall within the scope of disclosure and protection of this application.
[0041] Furthermore, the terms "first," "second," "S100," "S200," etc., used in the specification, claims, and drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in any order other than those described herein. At the same time, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. Unless otherwise expressly specified and limited, the terms "set," "arranged," "installed," "connected," and "linked" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to the internal connection of two components or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this case based on the specific circumstances and in conjunction with the prior art.
[0042] It should be noted that this application supports standard file systems, streaming data, and Scatter-Gather DMA, including but not limited to the above.
[0043] like Figure 1 This is a schematic diagram of the overall structure of an embedded storage device for high-speed streaming data according to this application. This component relies on hardware and software collaboration. Specifically, it includes: a peripheral device, a storage management module, a hardware and software interaction module, a file management module, and a buffer management module. The peripheral device is used to implement streaming data transmission and storage functions. The storage management module, in the form of a block device driver, controls the read and write operations between the peripheral device cache and the storage medium. The hardware and software interaction module is used to interact with the peripheral device, instructing it to complete streaming data transmission and processing. The file management module stores a mapping table between streaming data channels and stored files, associating streaming data channels with files. The buffer management module manages the peripheral device cache, implementing the mapping from processor memory to the peripheral device cache.
[0044] Specifically, the file management module interacts directly with the user, receiving write or read requests. It generates I / O requests and sends them to the virtual file layer, which processes and generates several block I / O requests, handing them over to the underlying block device driver. The storage management module receives the I / O requests through the block device driver, processes them, generates custom command packets, and adds them to the command queue. The hardware / software interaction module submits work requests to peripheral devices, instructing them to store data. The peripheral devices access the storage medium, move the data to the designated location, generate a response packet upon completion, and notify the hardware / software interaction module via a completion signal. The hardware / software interaction module processes the completion signal and notifies the storage management module to parse the response packet, returning the response result to the file management module. The buffer management module maps processor memory to peripheral device caches during system initialization and sends the address mapping table to the storage management module. The storage management module uses the address mapping table to determine the buffer for each I / O request.
[0045] The file management module is designed and implemented by a Linux application. Its main task is to associate streaming data channels with files, receive user requests, and generate corresponding I / O requests.
[0046] The standard file read / write interface mentioned in this example is an interface provided by the file system, which adopts the POSIX standard and completes file operations through operating system calls. This approach also facilitates the subsequent construction of network file systems.
[0047] The storage management module is designed and implemented using the Linux kernel block device driver, and is divided into a request processing module and an address mapping module. The main function of the request processing module is to parse I / O requests issued by the file system, generate command packets, and place them in a command queue. The main function of the address mapping module is to store the address mapping table of peripheral device caches. After obtaining the discrete addresses in the I / O requests, it manages the address buffer through a custom linked list.
[0048] The I / O requests mentioned in this example are processed by the file system and then passed to the block device driver in the form of block I / O (bio). The bio is a structure that the storage management module relies on to generate command packets. Key information in it includes: file read / write direction, block device address, data buffer address, and data size.
[0049] The command and response packets mentioned in this example have the same format, such as... Figure 2 As shown, the specific parameters include: command packet number, Phase bit, read / write flag bit, storage medium offset length, Sg list base address, Sg list length, Sg list type, and peripheral device memory base address. The Phase bit, filled by the peripheral device, indicates whether the command packet is valid.
[0050] The Sg list mentioned in this example is a custom linked list that stores several sg descriptors to manage discrete buffer addresses for parsing and processing by the SG DMA controller. Each segment of the Sg list is specified to be 255 characters long. If an sg descriptor is not the last descriptor in the current sg_list, it stores the address and length of the corresponding data segment; otherwise, the sg descriptor is used to point to the next sg_list segment. The SG DMA controller parses the Sg list according to this rule.
[0051] The address mapping module iterates through the bio requests, parses the data address of each page, generates discrete data buffer addresses, and stores them in the Sg list.
[0052] Communication between the hardware / software interaction module and peripherals is achieved through the doorbell register and interrupts. Command packets and response packets are managed through command queues and response queues, respectively.
[0053] The command queue and response queue mentioned in this example are managed through a FIFO queue. The command queue is filled by software and read by peripheral devices; the response queue is filled by peripheral devices and read by software.
[0054] The main tasks of the buffer management module are to allocate buffers, perform address mapping between processor memory and peripheral device memory, and manage the buffer locations for I / O requests.
[0055] In this example, the buffer design relies on the direct I / O mode. The direct I / O mode can skip the page cache and directly access the hard disk, reducing CPU memory copying. At the same time, removing the cache allows the application to directly manage the address of the buffer, making it easier for the application to complete address mapping.
[0056] The processor memory mentioned in this example is managed by the processor and is used to store and process data from the operating system, applications, and other tasks that need to be executed in the processing system. Data in I / O requests obtained by block device drivers is stored here.
[0057] The peripheral device memory mentioned in this example is used by peripheral devices to accelerate computing and data processing. It is more efficient for peripheral devices to write to disk from this address.
[0058] The address mapping mentioned in this example mainly includes two address replacement processes. The first address replacement is to replace the physical address of the peripheral device memory containing data with the virtual address of the processor memory. The second address replacement is to map the physical address of the processor memory to the physical address of the peripheral device memory. The specific implementation process is as follows: a large block of memory with the size of a kernel page is allocated in user space to obtain the discrete physical addresses of the processor memory corresponding to the memory. These physical addresses are then mapped to the physical addresses on the peripheral device memory. The processor memory data block with the smallest page size of the operating system is used to map the data block on each peripheral device memory.
[0059] The primary task of peripheral devices is to store streaming data into the storage medium. They move streaming data according to the command packets of the storage management module, fill in response packets, and return interrupts to complete a storage process.
[0060] Corresponding to the embedded storage device for high-speed streaming data in this application, this application also provides an embedded storage method for high-speed streaming data, such as... Figure 3 As shown, the steps include:
[0061] S100: The buffer management module allocates the processor memory physical address and the peripheral device cache physical address, generates an address mapping table, and hands it over to the address mapping module in the storage management module;
[0062] S200: The file management module generates a write I / O request, which is processed by the file system and then passed to the storage management module;
[0063] S300: The request processing module receives block I / O requests issued by the file system, processes them, generates corresponding command packets, and fills them into the command queue for execution.
[0064] S400: The software and hardware interaction module submits work requests to the peripheral device and notifies the peripheral device of the number of work requests sent through the doorbell register;
[0065] S500: Peripheral device parses command packets and moves streaming data from the buffer to the storage medium;
[0066] S600: After the transfer is completed, the peripheral device generates a response packet and fills it into the response queue, and returns a completion signal to the software and hardware interaction module;
[0067] S700: The hardware and software interaction module receives a completion signal, reads and parses the response packet returned by the peripheral device;
[0068] S800: The file management module monitors the file writing status and stops writing data to the file when the file writing fails or the file is full.
[0069] The specific structure and logic of the modules in the above steps are similar to those of the aforementioned embedded storage devices for high-speed streaming data.
[0070] In step S100, the block device driver obtains the relevant register address, interrupt number, and interrupt information through the platform driver and device tree.
[0071] like Figure 4 The request processing module in step S300 further includes the following steps:
[0072] S301: File system calls the block device driver request handling function;
[0073] S302: The block device driver parses the bio to obtain the processor memory physical address of the data buffer. Based on the address mapping table issued by the address mapping module, it determines the actual address of the data buffer. If the data address is in the mapping table, it performs address mapping from processor memory to peripheral memory and uses the corresponding peripheral device memory address as the data buffer. Otherwise, it does not perform mapping and directly uses the passed processor memory address as the data buffer.
[0074] S4303: Traverse bio, parse out each bio_vec structure, obtain the physical address and length of each page buffer, and perform address mapping according to the selected address buffer;
[0075] S304: Fill the address buffer information into the Sg list sequentially;
[0076] S305: Combine the bio information to generate and fill in the command package, and put the command package into the command queue.
[0077] In step S303, the bio_vec structure is used to describe all the memory corresponding to this bio request. It may not always be in a single page, so a vector description is required, which specifically includes the page pointer where the data segment is located, the length of the data segment, and the offset of the data segment in the page.
[0078] In step S400, the hardware / software interaction module notifies the peripheral device to retrieve the command packet by updating the doorbell register. The doorbell register stores the position of the tail of the command queue. The peripheral device compares the tail of the command queue with the head of the queue; if they are not equal, it retrieves the command packet.
[0079] In step S500, the peripheral device moves command packets according to the head position of the command queue, and updates the head position of the command queue before the moving begins. After the moving is completed, the peripheral device generates a response packet, fills it into the response queue, and updates the tail position of the response queue.
[0080] In step S700, the software and hardware interaction module also obtains the tail position of the response queue based on the doorbell register. If the positions of the head and tail of the response queue are not equal, the software and hardware interaction module retrieves the response packet.
[0081] In summary, the embedded storage device and method for high-speed streaming data provided in this application employs a hardware / software memory address mapping + address replacement scheme and designs a custom block device driver. Streaming data does not need to pass through processor memory, reducing the number of memory copies and lowering processor resource consumption, thereby improving the efficiency of streaming data storage. Furthermore, compared to existing technologies, this solution fully utilizes the standard file system, facilitating user-level file access and the construction of network file systems. Moreover, compared to existing technologies, this solution supports different processor architectures, offering strong program portability.
[0082] The preferred embodiments disclosed above are merely illustrative of this application. These preferred embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the content of this specification. The selection and detailed description of these embodiments in this specification are intended to better explain the principles and practical applications of this application, thereby enabling those skilled in the art to better understand and utilize this application. This application is limited only by the claims and their full scope and equivalents. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.
[0083] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the above embodiments do not limit the present invention in any way, and all technical solutions obtained by equivalent substitution or equivalent transformation fall within the protection scope of the present invention.
Claims
1. An embedded storage device for high-speed streaming data, characterized in that, include: Peripheral devices are used to implement streaming data transmission and storage functions; The storage management module, in the form of a block device driver, controls the read and write operations between the peripheral device cache and the storage medium; The hardware and software interaction module is used to interact with the peripheral device and instruct the peripheral device to complete the streaming data transmission and processing; The file management module is used to store a mapping table between streaming data channels and stored files, thus associating streaming data channels with files. The buffer management module is responsible for managing peripheral device caches and implementing the mapping from processor memory to peripheral device caches.
2. The embedded storage device for high-speed streaming data according to claim 1, characterized in that, The storage function of the peripheral device supports bidirectional transmission of streaming data between the disk and the data buffer. The peripheral device's transmission function supports bidirectional transmission of streaming data between the external device and the data buffer.
3. The embedded storage device for high-speed streaming data according to claim 1, characterized in that, The storage management module includes: The address mapping module stores the address mapping table between the processor memory and the peripheral device memory. It manages the address buffer through a custom linked list for the scatter-gather DMA controller to parse and process. The request processing module is used to process I / O requests issued by the file system and generate command packages.
4. The embedded storage device for high-speed streaming data according to claim 1, characterized in that, The data interaction between the software and hardware interaction module and the peripheral device is implemented based on command queues and response queues, and the software and hardware communicate through doorbell registers and interrupts.
5. The embedded storage device for high-speed streaming data according to claim 4, characterized in that, The command queue and response queue are managed through a FIFO queue. The command queue is filled by software and read by peripheral devices, while the response queue is filled by peripheral devices and read by software.
6. The embedded storage device for high-speed streaming data according to claim 1, characterized in that, The file management module is responsible for scheduling I / O requests and controlling the storage direction, capacity, and disk access location of streaming data; The file management module monitors the data storage status of the streaming data in real time and returns corresponding errors to the user when errors occur.
7. The embedded storage device for high-speed streaming data according to claim 1, characterized in that, The buffer management module designs an address mapping table from processor memory to peripheral device cache using a page mapping method; The buffer management module allocates a large block of address space and obtains the corresponding processor memory physical address. It fills the physical address of the selected peripheral device cache into the corresponding address mapping table and sends it to the storage management module.
8. An embedded storage method for high-speed streaming data, characterized in that, The embedded storage method for high-speed streaming data, applied to any one of claims 1-7, comprises: S100: The buffer management module allocates the processor memory physical address and the peripheral device cache physical address, generates an address mapping table, and hands it over to the address mapping module in the storage management module; S200: The file management module generates a write I / O request, which is processed by the file system and then passed to the storage management module; S300: The request processing module receives I / O requests from the file system, processes them, generates corresponding command packets, and fills them into the command queue for execution. S400: The software and hardware interaction module submits work requests to the peripheral device and notifies the peripheral device of the number of work requests sent through the doorbell register; S500: Peripheral devices acquire and parse command packets, and move streaming data from the buffer to the disk; S600: After the transfer is completed, the peripheral device generates a response packet and fills it into the response queue, and returns a completion signal to the software and hardware interaction module; S700: The hardware and software interaction module receives a completion signal, reads and parses the response packet returned by the peripheral device; S800: The file management module monitors the file writing status and stops writing data to the file when the file writing fails or the file is full.
9. An embedded storage method for high-speed streaming data according to claim 8, characterized in that, In step S300, the address mapping module parses the I / O request to obtain the physical address of the I / O request buffer, and determines the actual physical address of the I / O request by combining it with the address mapping table. If the physical address of the buffer is in the address mapping table, it is replaced with the physical address of the corresponding peripheral device memory; otherwise, no address replacement is performed.