A linear acquisition and storage network integrated device and storage architecture

Abstract

This invention discloses a linear data acquisition and storage network integrated device and its storage architecture. The linear data acquisition and storage network integrated device includes a front-end acquisition module, a recording control module, and a back-end storage array, realizing data filtering, forwarding, and storage. The storage architecture includes multiple linear data acquisition and storage network integrated devices operating in a cluster mode. This invention modifies the user data transmission path, allowing data to go directly from the front-end acquisition module to the back-end storage array, bypassing performance bottlenecks and achieving high-speed recording. Furthermore, the real-time forwarding by the front-end acquisition module enables load balancing, achieving cluster mode, and enabling data content parsing and management for refined data management; simultaneously, real-time filtering at the front end provides a firewall effect, protecting device security.

Description

Technical Field

[0001] This invention belongs to the field of data storage technology, specifically relating to a linear acquisition and storage network integrated device and storage architecture. Background Technology

[0002] With the increasing demand for integrated development of shipboard networks and the growing need for information exchange and sharing among various equipment, the amount of network information is also increasing dramatically. This places higher demands on the bandwidth and analytical capabilities of 10GE / 40GE high-speed network data acquisition, analysis, and storage equipment. At the same time, it also faces the challenge of real-time storage of various types of data at mixed high and low speeds, as well as application scenarios such as real-time parsing and association management of some types of data content. These challenges bring even greater difficulties to high-speed network data acquisition, analysis, and storage equipment.

[0003] The existing high-speed network data acquisition, analysis, and storage technology architecture is based on CPU acquisition and storage. User data is collected by the front-end acquisition module and first transmitted to the recording control module, and then transmitted by the recording control module to the back-end storage array. However, this architecture has the following problems in practical applications:

[0004] 1. Accuracy issues: In this storage architecture, data packets are first cached in the network card, and then a certain number of data packets are aggregated and transferred to the CPU's DDR via DMA. The arrival timestamp of the data packets is added by the kernel, and there is a certain delay in the time information. At the same time, due to the real-time problem of CPU scheduling, accurate scheduling cannot be guaranteed. In other words, the overall accuracy of the timestamp of collected network data packets is not high.

[0005] II. Performance Issues: During high-speed network data acquisition, the network protocol stack implementation limits the need to copy data from the kernel to the application; during data storage, data needs to be copied from the application layer to the kernel layer. These two data copies, especially during high-speed acquisition and storage, can cause CPU capabilities and DDR main memory access bandwidth to become performance bottlenecks along the entire data path, leading to packet loss. Due to power consumption and specific application environments, high-speed recording systems often employ embedded industrial control CPU platforms for acquisition and storage. These platforms have relatively limited processing power and memory access bandwidth, making it difficult for existing acquisition and recording technologies to adapt to all high-speed data recording scenarios.

[0006] III. Expansion Issues: The equipment has poor scalability. When the CPU of the recording control module cannot support the business, the problem cannot be solved by quickly adding more recording control modules. Summary of the Invention

[0007] One of the objectives of this invention is to provide a linear data acquisition and storage network integrated device, which is a network integrated product that integrates high-speed data acquisition and storage, load balancing, and real-time processing.

[0008] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0009] A linear acquisition and storage network integrated device, comprising a front-end acquisition module, a recording and control module, and a back-end storage array, wherein:

[0010] The front-end acquisition module is used to parse the acquired data packets and perform filtering, forwarding, and / or storage operations on the data packets according to the parsing results and configuration rules, wherein:

[0011] When the front-end acquisition module performs a filtering operation, it performs the following steps: filtering data packets based on the filtering mode determined by the parsing results and configuration rules; if the current data packet is filtered and discarded, the current operation ends and waits for the arrival of the next data packet; if the current data packet is not filtered and discarded, a forwarding operation and / or a storage operation is performed.

[0012] When the front-end acquisition module performs the forwarding operation, it performs the following steps: forwarding non-high-speed data in the data packet to the recording control module in a real-time transparent transmission manner, and then the recording control module transmits it to the back-end storage array for disk storage.

[0013] When the front-end acquisition module performs storage operations, it performs the following steps: caches the high-speed data in the data packet, and notifies the recording control module when the cached high-speed data reaches a preset data volume. Under the control of the recording control module, the back-end storage array directly reads the cached high-speed data from the front-end acquisition module and stores it on disk.

[0014] Several alternative methods are provided below, but they are not intended as additional limitations on the overall solution above. They are merely further additions or optimizations. Provided there are no technical or logical contradictions, each alternative method can be combined individually with respect to the overall solution above, or multiple alternative methods can be combined with each other.

[0015] Preferably, the configuration rules include a preset filtering mode based on the five-tuple information of the data packet. The filtering mode includes either white or black and neither black or white. The five-tuple information includes source IP, destination IP, source port, destination port, and protocol type.

[0016] Preferably, the front-end acquisition module includes an FPGA and a DDR cache unit. When the cached high-speed data reaches a preset data volume, the front-end acquisition module notifies the recording control module to perform the following operations:

[0017] An interrupt is generated to the recording control module, and the starting address of the high-speed data in the DDR cache unit, the effective length of the data, and the logical channel ID are sent to the recording control module. The high-speed data in the data packet is cached in the DDR cache unit.

[0018] Preferably, under the control of the recording control module, the back-end storage array directly reads the cached high-speed data from the front-end acquisition module and stores it on disk, performing the following operations:

[0019] The back-end storage array receives the starting address, effective length, and management logic address of the high-speed data in the DDR cache unit sent by the recording control module, initiates DMA, and moves the high-speed data in the DDR cache unit to the block address corresponding to the management logic address in the back-end storage array according to the starting address and effective length of the high-speed data in the DDR cache unit, thus completing the data write-to-disk operation. The management logic address is obtained by the recording control module based on the logical channel ID.

[0020] The linear acquisition and storage network integrated device provided by this invention adds a new network path from the front-end acquisition module to the recording control module. Based on the new hardware framework, the user data transmission path is modified so that data goes directly from the front-end acquisition module to the back-end storage array. The recording control module is only responsible for scheduling and management, avoiding performance bottlenecks and achieving high-speed recording. By adding the new network path, the front-end acquisition module forwards non-high-speed acquired data and commands to the recording control module in real time, where the operating system of the recording control module handles the network protocol stack processing.

[0021] The second objective of this invention is to provide a storage architecture that is flexible, efficient, and highly scalable.

[0022] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0023] A storage architecture includes multiple linear acquisition and storage network integrated devices operating in a cluster mode. Each linear acquisition and storage network integrated device includes a front-end acquisition module, a recording control module, and a back-end storage array, wherein:

[0024] The front-end acquisition module is used to parse the acquired data packets and perform filtering, forwarding, and / or storage operations on the data packets according to the parsing results and configuration rules, wherein:

[0025] When the front-end acquisition module performs a filtering operation, it performs the following steps: filtering data packets based on the filtering mode determined by the parsing results and configuration rules; if the current data packet is filtered and discarded, the current operation ends and waits for the arrival of the next data packet; if the current data packet is not filtered and discarded, a forwarding operation and / or a storage operation is performed.

[0026] When the front-end acquisition module performs a forwarding operation, it performs the following steps: forwarding non-high-speed data in the data packet to the recording control module in a real-time transparent transmission manner, and then the recording control module transmits it to the back-end storage array for disk storage; at the same time, based on the forwarding mode determined by the parsing results and configuration rules, it forwards non-high-speed data and / or high-speed data to the front-end acquisition module of the parallel linear acquisition and storage network integrated device for storage.

[0027] When the front-end acquisition module performs storage operations, it performs the following steps: caches the high-speed data in the data packet, and notifies the recording control module when the cached high-speed data reaches a preset data volume. Under the control of the recording control module, the back-end storage array directly reads the cached high-speed data from the front-end acquisition module and stores it on disk.

[0028] Preferably, the configuration rules include a preset filtering mode based on the five-tuple information of the data packet. The filtering mode includes either white or black and either black or white. The five-tuple information includes source IP, destination IP, source port, destination port and protocol type.

[0029] The configuration rules also include a forwarding mode preset based on the five-tuple information of the data packet, the forwarding mode including round-robin and dynamic load balancing.

[0030] Preferably, the front-end acquisition module includes an FPGA and a DDR cache unit. When the cached high-speed data reaches a preset data volume, the front-end acquisition module notifies the recording control module to perform the following operations:

[0031] An interrupt is generated to the recording control module, and the starting address of the high-speed data in the DDR cache unit, the effective length of the data, and the logical channel ID are sent to the recording control module. The high-speed data in the data packet is cached in the DDR cache unit.

[0032] Preferably, under the control of the recording control module, the back-end storage array directly reads the cached high-speed data from the front-end acquisition module and stores it on disk, performing the following operations:

[0033] The back-end storage array receives the starting address, effective length, and management logic address of the high-speed data in the DDR cache unit sent by the recording control module, initiates DMA, and moves the high-speed data in the DDR cache unit to the block address corresponding to the management logic address in the back-end storage array according to the starting address and effective length of the high-speed data in the DDR cache unit, thus completing the data write-to-disk operation. The management logic address is obtained by the recording control module based on the logical channel ID.

[0034] The storage architecture provided by this invention expands the direct data recording channel by redirecting the file access DMA address to adapt to high-speed recording and high-precision timestamp timing control application scenarios. According to the configuration rules, the front-end acquisition module forwards data in real time to achieve load balancing capability, reaching a cluster mode, enabling real-time parsing and processing of data content, and achieving fine-grained data management. At the same time, real-time filtering is performed at the front end to achieve the effect of a firewall and protect device security. Attached Figure Description

[0035] Figure 1 This is a schematic diagram of the structure of the linear acquisition and storage network integrated device of the present invention;

[0036] Figure 2 This is a schematic diagram of the storage architecture of the present invention. Detailed Implementation

[0037] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0038] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to limit the invention.

[0039] To overcome the problems of accuracy, performance, and scalability in existing high-speed network data acquisition, analysis, and storage technology architectures, a linear acquisition and storage network integrated device is proposed in one embodiment.

[0040] like Figure 1 As shown, the linear acquisition and storage network integrated device in this embodiment includes a front-end acquisition module, a recording control module, and a back-end storage array. The front-end acquisition device can be an FPGA and corresponding DDR memory (referred to as a DDR cache unit) for external data acquisition and caching, such as in a camera device. The read / write control device is typically a central processing unit (CPU) with a network card and corresponding DDR main memory (referred to as a system DDR main memory unit), such as in a computer device. The back-end storage can be an NVMe solid-state drive for reading and caching data, such as in a video recorder device.

[0041] In this embodiment, data interaction between the front-end acquisition module and the back-end storage array, and between the recording control module and the back-end storage array, is based on a data exchange module. The front-end acquisition module and the recording control module interact directly through the CPU's network card. The data exchange module is a PCIe switch, providing a high-speed data channel.

[0042] In this embodiment, the front-end acquisition module is used to parse the acquired data packets and perform filtering, forwarding and / or storage operations on the data packets according to the parsing results and configuration rules.

[0043] In this embodiment, the configuration rules include preset filtering modes based on the five-tuple information of the data packets. The filtering modes include either a black-and-white or black-and-white approach. The five-tuple information includes the source IP, destination IP, source port, destination port, and protocol type. In this embodiment, the filtering mode for each data stream is pre-defined in the configuration rules based on the five-tuple information.

[0044] Based on the application scenario, the following two working modes can be set:

[0045] 1) Neither whitelist nor blacklist mode: Only a whitelist is configured. Data packets in the whitelist are stored or forwarded according to the configured rules. If no whitelist is configured, the data packets are discarded by default. The default working mode of the device is the neither whitelist nor blacklist mode. In this working mode, the FPGA acts as a device firewall, which plays an isolation role and protects the device security.

[0046] 2) Black-and-white mode: Only a blacklist is configured, i.e. the list of filtered data packets. Data packets in the blacklist are discarded, and other data packets are stored or forwarded according to the configured rules; if no configuration is made, the default real-time transparent transmission mode is used for forwarding.

[0047] In this embodiment, when the front-end acquisition module (specifically, it may be executed by an FPGA) performs data filtering, it performs the following steps: filtering data packets based on the filtering mode determined by the parsing results and configuration rules; if the current data packet is filtered and discarded, the current operation ends and waits for the arrival of the next data packet; if the current data packet is not filtered and discarded, forwarding and / or storage operations are performed.

[0048] For data packets that have been filtered and retained, the FPGA performs data forwarding for non-high-speed data and command execution, and performs data storage for high-speed data.

[0049] In this embodiment, when the front-end acquisition module performs a forwarding operation, it executes the following steps: Non-high-speed data in the data packet is forwarded to the recording control module in a real-time transparent transmission manner, and then transmitted by the recording control module to the back-end storage array for disk storage. It is easy to understand that, in addition to non-high-speed data, the front-end acquisition module can also forward various commands to the recording control module.

[0050] When the front-end acquisition module of this embodiment performs storage operations, it performs the following steps: caching the high-speed data in the data packet, and notifying the recording control module when the cached high-speed data reaches a preset data volume, and the back-end storage array directly reads the cached high-speed data from the front-end acquisition module and stores it on disk under the control of the recording control module.

[0051] Compared to traditional storage technology architectures, the linear acquisition and storage network integrated device in this embodiment adds an extra network path from the front-end acquisition module to the recording control module for transmitting non-high-speed data. According to the configuration allocation rules, the front-end acquisition module can acquire and directly store data to the back-end storage array, thereby realizing that the data storage path does not pass through the recording control module. By modifying the data transmission path, the recording control module is only responsible for scheduling and management, avoiding performance bottlenecks and achieving the goal of high-speed recording.

[0052] It should be noted that the filtering, forwarding and storage operations in this embodiment can all be executed, or only some of the operations can be selected to be executed according to the actual application scenario. For example, the forwarding operation may not be executed, and the entire device may not process non-high-speed data, etc.

[0053] For ease of description, this embodiment uses a front-end acquisition module including an FPGA and a DDR cache unit as an example.

[0054] In this embodiment, when the FPGA performs data storage, it stores the collected high-speed data in the DDR cache unit. When it detects that the amount of high-speed data has reached the amount of data that can be transferred, it generates an interrupt to the recording control module and sends the starting address of the high-speed data in the DDR cache unit, the effective length of the data, and the logical channel ID to the recording control module.

[0055] The recording control module responds to the interrupt generated by the FPGA, receives the starting address of the high-speed data in the DDR cache unit, the effective length of the data, and the logical channel ID, obtains the management logical address based on the logical channel ID, and submits IO commands to the back-end memory array to transmit the starting address of the high-speed data in the DDR cache unit, the effective length of the data, and the management logical address to the back-end memory array.

[0056] After receiving the IO command submitted by the recording control module, the back-end storage array obtains the corresponding physical storage address based on the received management logic address and starts DMA. Based on the starting address of the high-speed data in the DDR cache unit and the effective length of the data, the array moves the data to be stored in the DDR cache unit to the corresponding physical storage address of the back-end storage array through the PCIe Switch bus addressing mode, thus completing the data storage and disk writing.

[0057] In traditional technical solutions, data collected by the front-end acquisition device is cached, then read into the memory of the recording control module before being written to the back-end storage array. The technical solution of this application directly writes the data collected by the front-end acquisition device from its cache to the back-end storage array, reducing the performance and bandwidth burden on the system in the read / write control device.

[0058] In traditional technical solutions, the back-end storage array reads data from the recording control module by initiating DMA. The DMA description includes three elements: the bus address of the data's location in system memory, the length of the data, and the starting block number of the storage bank where the data will be written.

[0059] In the existing architecture, the recording control module consists of application software implemented by a software program installed on the CPU, and a file system and storage driver provided by the kernel program in the recording control module's operating system. The general file system standard writes data to disk, and the DMA descriptor indicates that the file data is in the system main memory. This data is then retrieved from the system's DDR main memory by the storage device's DMA engine. However, this application initializes the starting address of high-speed data in the DDR cache unit into the DMA descriptor, allowing the storage device to directly write high-speed data from the front-end acquisition module to the disk without going through the recording control module.

[0060] In this embodiment, the existing file system and storage driver are extended while maintaining all the original features of the file system and storage driver. An additional dedicated interface for file extension is defined for application software to call. The specific extension operation is as follows:

[0061] A. A dedicated interface for the new file system: This dedicated interface is used by application software to pass the starting address of high-speed data in the DDR cache unit and the effective length of the data to the file system. This new dedicated interface can be implemented using the IOCTL system call, i.e., by adding a custom IOCTL command to pass parameters from the application layer to the file system layer in the operating system kernel.

[0062] B. File System Extension Functionality: This section describes the implementation of the IOCTL command response function for adding new files to the file system. The specific content of the response function implementation includes:

[0063] Upon receiving the valid data length provided by the file system's dedicated interface, the system calls the file system's regular block allocation function to allocate logical blocks according to the read / write size; combined with the starting address of the high-speed data within the DDR cache unit, a special BIO (Block Input Output) is created; a custom tag is added to the special BIO, and it is submitted to the block device layer via the regular path.

[0064] C. Storage driver extension function: When receiving BIO provided by the upper layer, a new tag judgment logic is added during processing. If it is a regular BIO, the original processing method is retained, that is, virtual address is converted into bus address. If it is a special BIO with a custom tag, the regular address conversion step is skipped, and the starting address of the high-speed data passed down from the upper layer is directly set in the DMA descriptor in the DDR cache unit.

[0065] In this embodiment, the application software calls the file system's dedicated interface to pass parameters to the file system. The file system responds, creates special block read / write operations, and submits the special block read / write operations to the storage driver. After receiving the special block read / write operations, the storage driver sets the starting address of the high-speed data in the DDR cache unit and the effective length of the data in the parameters into the DMA descriptor, encapsulates it into a read / write command, and submits it to the storage device.

[0066] It is readily understood that in this embodiment, the application software, file system, and storage driver in the recording control module directly transmit the starting address and effective length of the high-speed data sent by the FPGA to the back-end storage array within the DDR cache unit. The back-end storage array then directly retrieves the data from the FPGA and writes it to disk. The logic of the recording control module obtaining the management logic address based on the logic channel, and the back-end storage array converting the management logic address to the corresponding physical storage address, are all existing conventional processing logic and will not be elaborated upon in this embodiment.

[0067] Based on the application software, file system, and storage driver in the recording control module, a specific example of how a linear acquisition and storage network integrated device stores high-speed data is as follows:

[0068] 1) When the amount of real-time computing data in the FPGA reaches a portable level, the application software is notified via an interrupt. The parameters passed to the application software include: the starting address of the high-speed data in the DDR cache unit, the effective length of the data, and the logical channel ID.

[0069] 2) The application software first parses the parameters, then finds the file corresponding to the channel based on the logical channel ID, allocates the starting block number of the storage body to be written, obtains the management logical address, and finally calls the file system's dedicated interface to pass the starting address of the high-speed data in the DDR cache unit and the effective length of the data to the file system.

[0070] 3) The file system forms a write operation based on the management logical address. The storage driver submits an I / O command to the storage and transmits parameters including the starting address of the high-speed data in the DDR cache unit and the management logical address.

[0071] 4) The memory receives the IO command, starts DMA, and moves the data in the DDR cache unit to the corresponding block address through the PCIE switch bus addressing mode.

[0072] 5) The application software creates index information and writes it to the storage in real time. The data flow from the system's DDR main memory unit to the storage ensures that the index information and the data remain consistent.

[0073] In this embodiment, the existing data transmission path of the acquisition and storage architecture is modified. The original data flow of first moving network data from the DDR cache unit to the system DDR main memory unit, and then from the system DDR main memory unit to the storage unit, is changed to a data flow of directly moving network data from the DDR cache unit to the storage unit, thereby realizing high-speed data acquisition and recording.

[0074] This embodiment adds a network path to the traditional acquisition and storage architecture. The FPGA forwards non-high-speed acquired data and commands to the CPU in real time, and the CPU's operating system handles the network protocol stack processing. The application software only needs to be aware of user data and does not need to pay attention to the protocol stack, thus reducing software complexity. At the same time, based on the hardware framework, the data transmission path of the recording control module is changed by extending the file system and storage driver. Data goes directly from the FPGA to the storage, and the CPU is only responsible for scheduling and management, avoiding performance bottlenecks and thus achieving the goal of high-speed recording.

[0075] In another embodiment, such as Figure 2 As shown, a storage architecture is also provided, which includes multiple linear acquisition and storage network integrated devices in cluster operation mode. Each linear acquisition and storage network integrated device includes a front-end acquisition module, a recording control module, and a back-end storage array, wherein:

[0076] The front-end acquisition module is used to parse the acquired data packets and perform filtering, forwarding, and / or storage operations on the data packets according to the parsing results and configuration rules, wherein:

[0077] When the front-end acquisition module performs a filtering operation, it performs the following steps: filtering data packets based on the filtering mode determined by the parsing results and configuration rules; if the current data packet is filtered and discarded, the current operation ends and waits for the arrival of the next data packet; if the current data packet is not filtered and discarded, a forwarding operation and / or a storage operation is performed.

[0078] When the front-end acquisition module performs a forwarding operation, it performs the following steps: forwarding non-high-speed data in the data packet to the recording control module in a real-time transparent transmission manner, and then transmitting it to the back-end storage array for disk storage by the recording control module; at the same time, based on the forwarding mode determined by the parsing results and configuration rules, forwarding non-high-speed data and / or high-speed data to the front-end acquisition module of the parallel linear acquisition and storage network integrated device.

[0079] When the front-end acquisition module performs storage operations, it performs the following steps: caches the high-speed data in the data packet, and notifies the recording control module when the cached high-speed data reaches a preset data volume. Under the control of the recording control module, the back-end storage array directly reads the cached high-speed data from the front-end acquisition module and stores it on disk.

[0080] In one specific embodiment, the configuration rule includes a preset filtering mode based on the five-tuple information of the data packet. The filtering mode includes either white or black and either black or white. The five-tuple information includes source IP, destination IP, source port, destination port, and protocol type.

[0081] The configuration rules also include a forwarding mode preset based on the five-tuple information of the data packet, the forwarding mode including round-robin and dynamic load balancing.

[0082] In one specific embodiment, the front-end acquisition module includes an FPGA and a DDR cache unit. When the cached high-speed data reaches a preset data volume, the front-end acquisition module notifies the recording control module to perform the following operations:

[0083] An interrupt is generated to the recording control module, and the starting address of the high-speed data in the DDR cache unit, the effective length of the data, and the logical channel ID are sent to the recording control module. The high-speed data in the data packet is cached in the DDR cache unit.

[0084] In one specific embodiment, the back-end storage array, under the control of the recording control module, directly reads the cached high-speed data from the front-end acquisition module and writes it to disk for storage, performing the following operations:

[0085] The back-end storage array receives the starting address, effective length, and management logic address of the high-speed data in the DDR cache unit sent by the recording control module, initiates DMA, and moves the high-speed data in the DDR cache unit to the block address corresponding to the management logic address in the back-end storage array according to the starting address and effective length of the high-speed data in the DDR cache unit, thus completing the data write-to-disk operation. The management logic address is obtained by the recording control module based on the logical channel ID.

[0086] For specific limitations on the storage architecture, please refer to the limitations of the linear acquisition and storage network integrated equipment, which will not be elaborated here.

[0087] In the storage architecture, based on configuration rules, the FPGA distributes network data load to other devices, thus playing a load-sharing role. With current FPGA resources, it can handle two incoming network ports and distribute the load across 16 outgoing network ports according to the traffic configuration rules, thereby achieving horizontal scaling.

[0088] Since the storage architecture of this embodiment includes multiple linear acquisition and storage network integrated devices in cluster working mode, that is, the storage architecture of this embodiment implements cluster working mode, the forwarding mode of the linear acquisition and storage network integrated devices in the storage architecture of this embodiment also includes forwarding data to the parallel linear acquisition and storage network integrated devices to achieve load balancing and achieve cluster working mode, similar to a network load balancer (F5).

[0089] Based on the application scenario, the following two working modes can be set:

[0090] 1) Polling: Based on the device source IP and port, poll to different devices, ensuring that the same service is processed on the same device.

[0091] 2) Dynamic load: Based on the real-time load of the backend devices (FPGA or CPU), the application software can dynamically modify the FPGA forwarding rules to achieve dynamic load balancing; at the same time, based on the working status of the devices, the application software can dynamically modify the FPGA forwarding rules to enable devices to dynamically join or leave, thereby achieving dynamic expansion, i.e., cluster working mode.

[0092] When a single device's CPU cannot handle the entire workflow of high-speed data acquisition, parsing, and storage in real time, the traffic load can be reduced by dynamically expanding the device, enabling multiple devices to process in parallel, i.e., cluster mode. This allows for frame-level management of data content, i.e., refined data management. Data is processed according to its characteristics, with high-speed data being directly stored and medium- and low-speed data being forwarded (transparently transmitted) to the device's CPU in real time, thus achieving content-level management and storage of medium- and low-speed real-time data.

[0093] The storage architecture in this embodiment achieves high-speed storage in a simple way, reducing the dependence of high-speed storage on high-performance CPUs. At the same time, it retains the original characteristics of the file system; data files generated by high-speed storage can be accessed through standard file access interfaces, facilitating storage management. Depending on different application scenarios: high-speed data is managed at the content level, achieving a cluster working mode of dynamic load balancing through parallel processing of multiple devices according to configured forwarding rules; high-speed data is only collected and stored without real-time analysis, but rather analyzed post-hocly; medium- and low-speed data is managed at the content level through real-time forwarding according to configured rules; and the FPGA performs real-time filtering at the front end, achieving the effect of a device firewall and ensuring product security. These three advantages can be simultaneously achieved, resulting in a storage architecture with flexibility, high process efficiency, and strong scalability.

[0094] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0095] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the appended claims.

Claims

1. A storage architecture, characterized in that, The storage architecture includes multiple linear acquisition and storage network integrated devices operating in cluster mode. Each linear acquisition and storage network integrated device includes a front-end acquisition module, a recording and control module, and a back-end storage array, wherein: The front-end acquisition module is used to parse the acquired data packets and perform filtering, forwarding, and / or storage operations on the data packets according to the parsing results and configuration rules. The configuration rules include a preset forwarding mode based on the five-tuple information of the data packets. The forwarding mode includes round-robin and dynamic load balancing. The five-tuple information includes source IP, destination IP, source port, destination port, and protocol type, wherein: When the front-end acquisition module performs a filtering operation, it performs the following steps: filtering data packets based on the filtering mode determined by the parsing results and configuration rules; if the current data packet is filtered and discarded, the current operation ends and waits for the arrival of the next data packet; if the current data packet is not filtered and discarded, a forwarding operation and / or a storage operation is performed. When the front-end acquisition module performs a forwarding operation, it performs the following steps: forwarding non-high-speed data in the data packet to the recording control module in a real-time transparent transmission manner, and then the recording control module transmits it to the back-end storage array for disk storage; at the same time, based on the forwarding mode determined by the parsing results and configuration rules, it forwards non-high-speed data and / or high-speed data to the front-end acquisition module of the parallel linear acquisition and storage network integrated device for storage. When the front-end acquisition module performs storage operations, it performs the following steps: caches the high-speed data in the data packet, and notifies the recording control module when the cached high-speed data reaches a preset data volume. Under the control of the recording control module, the back-end storage array directly reads the cached high-speed data from the front-end acquisition module and stores it on disk.

2. The storage architecture as described in claim 1, characterized in that, The configuration rules also include a preset filtering mode based on the five-tuple information of the data packet, and the filtering mode includes either white or black and either black or white.

3. The storage architecture as described in claim 1, characterized in that, The front-end acquisition module includes an FPGA and a DDR cache unit. When the cached high-speed data reaches a preset data volume, the front-end acquisition module notifies the recording control module to perform the following operations: An interrupt is generated to the recording control module, and the starting address of the high-speed data in the DDR cache unit, the effective length of the data, and the logical channel ID are sent to the recording control module. The high-speed data in the data packet is cached in the DDR cache unit.

4. The storage architecture as described in claim 3, characterized in that, Under the control of the recording control module, the back-end storage array directly reads the cached high-speed data from the front-end acquisition module and stores it on disk, performing the following operations: The back-end storage array receives the starting address, effective length, and management logic address of the high-speed data in the DDR cache unit sent by the recording control module, initiates DMA, and moves the high-speed data in the DDR cache unit to the block address corresponding to the management logic address in the back-end storage array according to the starting address and effective length of the high-speed data in the DDR cache unit, thus completing the data write-to-disk operation. The management logic address is obtained by the recording control module based on the logical channel ID.

Images