Flow analysis board card based on DDR efficient storage
By combining FPGA and DDR storage modules in the traffic analysis board, dynamic partition storage and dynamic interrupt triggering mechanism are adopted, the problem of data loss in traditional boards in high traffic scenarios is solved, efficient data storage and real-time uploading is achieved, and the flexibility and practicality of network traffic analysis is improved.
Patent Information
- Application Number
- CN202510267106.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-03-07
- Publication Date
- 2025-06-27
AI Technical Summary
Traditional traffic analysis boards are difficult to upload and process all network data packets in real time in high traffic scenarios, resulting in data loss and inaccurate test results.
The traffic analysis board based on DDR efficient storage is adopted, combined with FPGA and DDR storage modules, and the dynamic partition storage method and dynamic interrupt triggering mechanism are used to achieve efficient data storage and upload.
Ensure that all received data frames can still be stored at high traffic peaks, avoid data loss, improve data transmission efficiency, ensure real-time data upload, and support traffic playback, protocol analysis and statistical analysis.
Smart Images

Figure CN120216444A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of network testers, and particularly relates to a traffic analysis board based on DDR high-efficiency storage. Background Art
[0002] With the rapid development of network technology, the scale and complexity of network traffic have been increasing continuously, posing higher requirements for the performance of traffic analysis boards. Traditional traffic analysis boards usually rely on the PCIE interface to interact with the host computer. However, in high-traffic scenarios, the low-version PCIE and the limited number of channels restrict the maximum bandwidth, resulting in the inability to upload and process all network data packets in real time. This limitation not only affects the real-time performance of the test but also may lead to data loss, thereby affecting the accuracy of the test results.
[0003] To solve the above problems, the existing technology usually adopts methods such as increasing the number of PCIE channels or improving the generation. However, these methods are still difficult to meet the requirements in extremely high-traffic scenarios. In addition, traditional network test boards usually do not have the ability to save each data frame, which limits their application in complex network environments.
[0004] In recent years, with the progress of storage technology, DDR memory has been widely used in various high-performance computing and data storage scenarios due to its characteristics of high bandwidth and low latency. However, how to combine DDR memory with network test boards to achieve efficient data storage and traffic analysis is still a technical problem to be solved urgently. Summary of the Invention
[0005] To solve the above problems existing in the prior art, the present invention provides a traffic analysis board based on DDR high-efficiency storage. The technical problems to be solved by the present invention are achieved through the following technical solutions:
[0006] The present invention provides a traffic analysis board based on DDR high-efficiency storage, including:
[0007] A power supply module, an SFP optical port transceiver, a clock module, an FPGA, and a DDR storage module; wherein,
[0008] The power supply module is used to supply power to each module in the traffic analysis board;
[0009] The SFP optical port transceiver is used to receive data frames in the network;
[0010] The clock module is used to provide a clock signal for the operation of the FPGA;
[0011] The FPGA is used to pack the data frames received by the SFP optical port transceiver into data units;
[0012] The DDR memory module is used to store the data units based on the dynamic partition storage method under the control of the FPGA;
[0013] The FPGA is further configured to convert the data units stored in the DDR memory module into reporting data packets in Pcap format; and upload the reporting data packets to the host computer connected to the traffic analysis board based on the dynamic interrupt trigger mechanism for traffic idle detection.
[0014] In an embodiment of the present invention, the FPGA includes:
[0015] An input sub-module, a storage control sub-module, a format conversion sub-module, and a PCIE reporting sub-module; wherein,
[0016] The input sub-module is configured to pack the data frames received by the SFP optical port transceiver into data units;
[0017] The storage control sub-module is configured to control the DDR memory module to store the data units;
[0018] The format conversion sub-module is configured to convert the data units stored in the DDR memory module into reporting data packets in Pcap format;
[0019] The PCIE reporting sub-module is configured to upload the reporting data packets to the host computer connected to the traffic analysis board based on the dynamic interrupt trigger mechanism for traffic idle detection.
[0020] In an embodiment of the present invention, the input sub-module packs the data frames into data units, including:
[0021] The input sub-module calculates the frame length of the data frame and records the reception time point as a time stamp, and uses the frame length and the time stamp as a frame description header;
[0022] Calculates the CRC checksum for the data frame to obtain the CRC checksum;
[0023] Adds the frame description header to the front of the data frame, adds the CRC checksum to the end of the data frame, obtains the data unit, and temporarily stores the data unit in its own buffer FIFO.
[0024] In an embodiment of the present invention, the DDR memory module stores the data units based on the dynamic partition storage method under the control of the FPGA, including:
[0025] The FPGA verifies the CRC checksum in the data unit and sends the complete data unit to the DDR memory module;
[0026] In the AXI write address channel of the DDR memory module, the burst transfer length is calculated according to the frame length of the data frame, and consecutive write addresses are generated.
[0027] According to the consecutive write addresses, based on the dynamic partition storage method, the data units are stored in the corresponding positions of the DDR memory module.
[0028] In an embodiment of the present invention, the DDR memory module includes:
[0029] A data frame storage area, a metadata storage area, and a management area; wherein,
[0030] The data frame storage area is used to store the data frames in the data units.
[0031] The metadata storage area is used to store the frame description header, timestamp, and CRC check code in the data units.
[0032] The management area is used to store the management information of the DDR memory.
[0033] In an embodiment of the present invention, the format conversion sub-module converts the data units stored in the DDR memory module into reported data packets in Pcap format, including:
[0034] The format conversion sub-module is a preset IP core. The format conversion sub-module reads the complete data frame from the data units stored in the DDR memory module.
[0035] The data frame is encapsulated in Pcap format to obtain reported data packets in Pcap format.
[0036] In an embodiment of the present invention, the Pcap format of the reported data packets includes:
[0037] A file header, a data packet record, a data packet header, and a data frame.
[0038] In an embodiment of the present invention, the PCIE reporting sub-module uploads the reported data packets to the host computer connected to the traffic analysis board based on the dynamic interrupt trigger mechanism of traffic idle detection, including:
[0039] After the test is completed or when the traffic is idle, an interrupt signal is sent to the host computer using the PCIE interface in the PCIE reporting sub-module, and the reported data packets to be uploaded are transmitted to the host computer at one time through the PCIE interface.
[0040] The beneficial effects of the present invention:
[0041] In the solution provided by the present invention, through the combination of the FPGA and the DDR storage module, the large capacity and high bandwidth characteristics of the DDR storage module are utilized to enable real-time caching of data frames in the network, ensuring that all received data frames can be stored even during high traffic peaks, thus avoiding data loss; the dynamic partition storage method is used to store data units, improving the writing efficiency and reducing the possibility of packet loss. The present invention uses a dynamic interrupt trigger mechanism to replace the traditional polling method, significantly improving the data transmission efficiency and ensuring the real-time nature of data upload. Before uploading data to the host computer, the data units are pre-converted into reported data packets in Pcap format. By performing data conversion in advance, not only the integrity and timing information of the data frames are retained, but also traffic playback, protocol parsing, and statistical analysis are supported, greatly enhancing the flexibility and practicality of network traffic analysis. Description of the Drawings
[0042] Figure 1 It is a schematic structural diagram of a traffic analysis board based on DDR efficient storage provided by an embodiment of the present invention;
[0043] Figure 2 It is a schematic diagram of the process of packing data frames into data units provided by an embodiment of the present invention;
[0044] Figure 3 It is a schematic diagram of the partition structure of the DDR storage module provided by an embodiment of the present invention;
[0045] Figure 4 It is a schematic diagram of the data structure of the reported data packet provided by an embodiment of the present invention;
[0046] Figure 5 It is a schematic diagram of the process of the PCIE reporting sub-module uploading the reported data packet provided by an embodiment of the present invention. Detailed Embodiments
[0047] The following describes the present invention in further detail with reference to specific embodiments, but the embodiments of the present invention are not limited thereto.
[0048] An embodiment of the present invention provides a traffic analysis board based on DDR efficient storage, as Figure 1 shown, which may include:
[0049] A power supply module, an SFP optical port transceiver, a clock module, an FPGA, and a DDR storage module; wherein,
[0050] The power supply module is used to supply power to each module in the traffic analysis board;
[0051] The SFP optical port transceiver is used to receive data frames in the network;
[0052] A clock module for providing a clock signal for the operation of the FPGA;
[0053] An FPGA for packing the data frames received by the SFP optical port transceiver into data units and performing other operations such as control and configuration;
[0054] A DDR storage module for storing data units based on the dynamic partition storage method under the control of the FPGA;
[0055] The FPGA is also used to convert the data units stored in the DDR storage module into reported data packets in Pcap format; and upload the reported data packets to the host computer connected to the traffic analysis board based on the dynamic interrupt trigger mechanism of traffic idle detection.
[0056] In the embodiment of the present invention, through the combination of the FPGA and the DDR storage module, the large capacity and high bandwidth characteristics of the DDR storage module are utilized to enable the data frames in the network to be cached in real time, ensuring that all received data frames can still be stored during high traffic peaks and avoiding data loss; the dynamic partition storage method is used to store data units, improving the writing efficiency and reducing the possibility of packet loss. The present invention uses the dynamic interrupt trigger mechanism to replace the traditional polling method, significantly improving the data transmission efficiency and ensuring the real-time nature of data upload. Before uploading the data to the host computer, the data units are pre-converted into reported data packets in Pcap format. By pre-converting the data, not only the integrity and timing information of the data frames are retained, but also traffic playback, protocol parsing, and statistical analysis are supported, greatly enhancing the flexibility and practicality of network traffic analysis.
[0057] For ease of understanding, each important module in the traffic analysis board based on DDR efficient storage proposed in the embodiment of the present invention will be introduced separately below.
[0058] The FPGA module includes:
[0059] An input sub-module, a storage control sub-module, and a format conversion sub-module; where
[0060] The input sub-module is used to pack the data frames received by the SFP optical port transceiver into data units;
[0061] The storage control sub-module is used to control the DDR storage module to store data units;
[0062] The format conversion sub-module is used to convert the data units stored in the DDR storage module into reported data packets in Pcap format;
[0063] The PCIE reporting sub-module is used to upload the reported data packets to the host computer connected to the traffic analysis board based on the dynamic interrupt trigger mechanism of traffic idle detection.
[0064] Specifically, for the process schematic diagram of packing data frames into data units, please refer to Figure 2 As shown, the input sub-module packs data frames into data units, which may include:
[0065] The input sub-module calculates the frame length of the data frame and records the reception time point as a timestamp, and takes the frame length and timestamp as the frame description header;
[0066] Calculate the CRC checksum for the data frame to obtain the CRC checksum;
[0067] Add the frame description header to the front of the data frame, add the CRC checksum to the end of the data frame to obtain the data unit, and temporarily store the data unit in its own buffer FIFO.
[0068] In a high-traffic network test scenario, data frame caching is the core link to achieve efficient data storage, and data integrity and correctness are the keys to ensuring the reliability of test results. To avoid data corruption caused by transmission errors or storage anomalies during data writing and reading, the present invention designs a complete data integrity guarantee mechanism. This mechanism adds a CRC checksum to the end of each data frame and combines the collaborative verification of the FPGA and the host computer to ensure data integrity and correctness. The detailed design of this mechanism is as follows:
[0069] The input sub-module in the FPGA uses a predefined CRC polynomial to calculate the data frame and generates a CRC checksum with a fixed length.
[0070] Attach the calculated CRC checksum to the end of the data frame to form a complete data unit. This step is completed before writing the data frame to the DDR storage module to ensure that each data frame stored contains verification information. By adding the verification information to the data unit, it is convenient to check and verify the data stored in the DDR storage module later.
[0071] By adding the frame description header to the front of the data frame and adding the CRC checksum to the end of the data frame, the data unit is obtained, and the data unit is temporarily stored in its own buffer FIFO, waiting for the data unit to be written into the DDR storage module.
[0072] For the process schematic diagram of packing data frames into data units provided by the embodiments of the present invention, please refer to Figure 2 As shown, in the embodiments of the present invention, each received data frame is written into the DDR memory on the board in real time through the customized storage control sub-module inside the FPGA to ensure the continuity and efficiency of data writing. Specifically, when a data frame is received at the FPGA port, a complete data unit is first generated, starting fromFigure 2 As can be seen, the frame length and timestamp are added to the front of the received data frame, and the CRC check code is added to the tail of the data frame. Subsequently, the data unit is sent to the buffer FIFO for temporary storage and waits to be written into the DDR storage module.
[0073] It can be understood that in high-traffic network test scenarios, the internal storage resources of the FPGA (such as BRAM and LutRAM) are usually very limited. Taking the K7-325T in the Xilinx Kintex-7 series as an example, this FPGA achieves a good balance among performance, resources, and price, but its internal BRAM resource is only 16 Mb, which consists of 440 36K Blocks. In applications such as network frame processing, image processing, and data sampling, the storage resources of the FPGA are often occupied in large quantities. For example, in scenarios such as asynchronous FIFOs required for crossing clock domains, call overheads of complex IP cores, and fast look-up table operations, BRAM is almost the only option. In the project evaluation stage, if it is found that the BRAM or LUT resources are insufficient, usually only a higher-end FPGA chip can be selected, which will lead to a significant increase in costs.
[0074] To solve this problem, in embodiments of the present invention, in scenarios such as network frame processing that can tolerate a certain storage-read latency, an off-chip DDR memory can be selected as the DDR storage module, significantly improving the data throughput capacity of the FPGA.
[0075] DDR storage module
[0076] Under the control of the FPGA, the DDR storage module stores data units based on the dynamic partition storage method, which can include:
[0077] The FPGA verifies the CRC check code in the data unit and sends the complete data unit to the DDR storage module;
[0078] The AXI write address channel in the DDR storage module calculates the length of the burst transmission according to the frame length of the data frame and generates consecutive write addresses;
[0079] According to the consecutive write addresses, based on the dynamic partition storage method, the data unit is stored in the corresponding position of the DDR storage module.
[0080] When writing the data frame into the DDR storage module, the FPGA will perform an integrity check on the data again to ensure that no errors occur during the writing process. Before the data is written into the DDR storage module through the AXI interface, the FPGA will recalculate the CRC check code and compare it with the check code appended to the frame tail. If the two are inconsistent, it indicates that the data may have been damaged during preprocessing or transmission. At this time, the FPGA will trigger an error handling mechanism and discard the data frame.
[0081] The data units stored in the buffer FIFO are transmitted to the DDR storage module through the AXI write channel, ensuring that each data frame is continuously stored and the addresses are consecutive. This design not only makes full use of the burst transmission characteristics of the DDR storage module, but also ensures the integrity and efficient caching and storage of data frames through the frame description header and the buffer FIFO mechanism, providing a reliable basis for subsequent traffic analysis and playback.
[0082] An embodiment of the present invention proposes an efficient storage design scheme based on an off-chip DDR storage module. The core idea of this scheme is: in scenarios such as network frame processing that can tolerate a certain storage - read latency, the DDR storage module is used as a large-capacity storage pool to significantly improve the data throughput capacity of the FPGA.
[0083] In the analysis of the storage resource bottleneck of the entire system, the frame staging queue is a typical scenario with relatively high requirements for storage capacity but relatively high tolerance for latency. For example: Network frames can stay in the storage queue for a certain period of time, as long as the correctness of the frames and the dequeue interval meet the requirements of IFS (Inter-Frame Spacing). It can be understood that the off-chip DDR memory as a storage medium has the following advantages:
[0084] Large-capacity storage: The capacity of the DDR memory is much larger than the BRAM inside the FPGA, which can support deeper queues, thereby improving the processing ability for burst traffic.
[0085] Multi-queue support: Each port can have independent queues, and even different queues can be allocated for traffic with different priorities to meet the requirements of complex network environments.
[0086] Cost-effectiveness: Using off-chip DDR memory can avoid the problem of replacing a higher-end FPGA chip due to insufficient BRAM resources, significantly reducing the hardware cost.
[0087] It can be understood that the physical characteristics of the DDR have problems such as read-write latency and low latency for adjacent addresses during page switching. In the embodiments of the present invention, a buffer FIFO is used before both the write and read interfaces to avoid the problem that data frames not processed in time due to excessive latency are flushed. By using the FIFO and handshaking mechanism, the outside is made unaware of the existence of latency. For the problem of low latency for adjacent addresses during page switching, by optimizing the address access mode, the address space is aligned to 8KB. When writing and reading, ensure that the address is only an integer multiple of 8KB, and use burst continuous access to calculate the frame length to obtain the burst size and merge it into 8K, so as to make the bubbles between each transmission as small as possible.
[0088] In a high-traffic network test scenario, the efficient management and partition planning of the DDR storage module are crucial to ensure that no overwriting or conflicts occur during the data storage and reading processes. Embodiments of the present invention guarantee the security and efficiency of data during the writing and reading processes through a refined partition planning design. Integer multiples of the frame length region are automatically divided according to the analyzed frame length data to meet the burst writing requirements.
[0089] To meet the data storage requirements in high-traffic scenarios, the DDR storage module is divided into multiple logical partitions, each with specific functions and uses. As shown in the DDR storage module, Figure 3 it can include:
[0090] a data frame storage area, a metadata storage area, and a management area; where
[0091] the data frame storage area is used to store the data frames in the data unit;
[0092] the metadata storage area is used to store the frame description header, timestamp, and CRC check code in the data unit;
[0093] the management area is used to store the management information of the DDR memory.
[0094] The data frame storage area is the main area of the DDR storage module and is used to store the data frames in the received data unit. This data frame storage area is further divided into multiple sub-areas, each corresponding to a port or priority queue, ensuring that data from different ports or priorities does not interfere with each other.
[0095] The metadata storage area is used to store metadata information, which corresponds one-to-one with the data frame storage area for easy and quick searching and parsing of frame information.
[0096] The management area is used to store the management information of the DDR memory, such as partition status, read / write pointers, error logs, etc. This management area is directly managed by the storage control sub-module in the FPGA to ensure the dynamic adjustment and efficient operation of the partition planning.
[0097] To achieve the efficient management of partition planning, a dedicated storage control sub-module is designed inside the FPGA, which is responsible for the allocation, release, and status monitoring of partitions. The specific mechanism is as follows:
[0098] Partition allocation: When the FPGA receives a data frame, the storage control sub-module will, according to information such as the port number and priority, store it in the address spaces of the data frame storage area and the metadata storage area. During the storage process, the storage control sub-module will ensure that the address ranges of each partition do not overlap to avoid data overwriting.
[0099] Partition status monitoring: The storage control sub-module monitors the status of each partition in real time, such as whether it is full, whether it is being read or written, etc. Through status monitoring, the storage control sub-module can dynamically adjust the partition allocation strategy to ensure that the storage and reading of high-priority data are prioritized.
[0100] Partition release: When the data in a certain partition is successfully uploaded to the host computer, the storage control sub-module marks the partition as the released state for subsequent data storage.
[0101] To prevent data from being overwritten or conflicting during the writing and reading processes, the embodiments of the present invention design a strict isolation mechanism as follows:
[0102] Writing isolation: When writing data to the DDR storage module, the storage control sub-module ensures that the writing operation is only performed in the data frame storage area and the metadata storage area, and the temporary buffer and the management area will not be written. In addition, the storage control sub-module checks the status of the target partition to ensure that the un-uploaded data will not be overwritten.
[0103] Reading isolation: During the data reading and uploading process, the storage control sub-module copies the data to be uploaded from the data frame storage area to the temporary buffer to ensure that the original data will not be overwritten by new writing operations. At the same time, the storage control sub-module locks the relevant partitions to prevent writing conflicts during the reading process.
[0104] It can be understood that the BRM cache of the traditional traffic analysis board card often uses FPGA and is very small, or directly uses PCIE transmission (packets will be lost under high traffic), and the application of conventional DDR only serves as a simple data cache pool without being optimized for the characteristics of network traffic, resulting in reduced reading and writing efficiency. To solve the above problems, the embodiments of the present invention adopt the dynamic block storage technology, dynamically adjust the size of the DDR storage block in this area according to the traffic characteristics (differentiated storage brought by the different lengths of TCP and UDP packets), and write the entire data block frame into the DDR through a single burst transmission, improving the writing efficiency and reducing the possibility of packet loss. The embodiments of the present invention adopt the FIFO prefetch mechanism, the DDR writing strategy based on traffic pattern prediction, to reduce the access delay; adopt the hardware-level verification mechanism, and integrate the CRC verification unit in the storage control sub-module to ensure the storage integrity.
[0105] Format conversion sub-module
[0106] The format conversion sub-module converts the data units stored in the DDR storage module into the reported data packets in Pcap format, which may include:
[0107] The format conversion sub-module is a preset IP core. The format conversion sub-module reads the complete network data frame from the data units stored in the DDR storage module;
[0108] Encapsulate the network data frame in Pcap format to obtain the reported data packet in Pcap format.
[0109] In network testing and analysis, converting the data frames captured by FPGA in the network into the Pcap format recognizable by the network analysis tool Wireshark is a key step to achieve traffic playback and protocol analysis. To complete this conversion, it is first necessary to understand the structures of the traditional Ethernet MAC frame format and the Pcap format, and design the corresponding conversion logic.
[0110] The traditional Ethernet MAC frame is the most basic unit in network communication. Its structure follows the IEEE802.3 standard. A typical Ethernet MAC frame can include the following parts:
[0111] Preamble: 7 bytes, used to synchronize the clock of the receiving end.
[0112] Start Frame Delimiter (SFD): 1 byte, indicating the start of the frame.
[0113] Destination MAC Address: 6 bytes, specifying the target device of the frame.
[0114] Source MAC Address: 6 bytes, identifying the sending device of the frame.
[0115] Length / Type field: 2 bytes, indicating the length of the frame or the upper-layer protocol type (such as: IPv4, IPv6).
[0116] Payload: 46 - 1500 bytes, containing the data of the upper-layer protocol (such as: IP packet, TCP segment, etc.).
[0117] Frame Check Sequence (FCS): 4 bytes, used for CRC check to ensure the integrity of the frame.
[0118] In the data frames captured by FPGA, the preamble and SFD are usually filtered out by hardware. Therefore, the actual stored data frames start from the destination MAC address and end at the FCS.
[0119] The Pcap format of the reported data packet, as Figure 4 shown, can include:
[0120] File header, packet record, data packet header, and data frame.
[0121] The Pcap (Packet Capture) format is a standard file format used by network analysis tools such as Wireshark to store network data frames. For the schematic diagram of the data structure of the reported data packet provided by the embodiments of the present invention, please refer to Figure 4 as shown in Figure 4 , it can be seen that a Pcap file consists of a file header and multiple packet records:
[0122] File header (Global Header): 24 bytes, containing metadata of the Pcap file, such as magic number (identifying the file format), timestamp precision, maximum frame length, etc.
[0123] Packet Record: Each packet record contains a packet header and the actual data frame.
[0124] Packet Header: 16 bytes, containing timestamp (seconds and microseconds), capture length (actual frame length), and original length (actual length of the frame).
[0125] Packet Data: Stores the actual network data frame, usually an Ethernet MAC frame.
[0126] On the FPGA side, the conversion logic from Ethernet MAC frame to Pcap format can be implemented in the following ways:
[0127] FPGA-side preprocessing: In the FPGA, before storing the data frame into the DDR storage module, a frame description header can be added to provide necessary information for subsequent conversion.
[0128] Pcap conversion module: Generates a file header and packet records according to the Pcap format, and converts the original data frame into the Pcap format. To improve efficiency and configurability, this module is encapsulated into an IP core, and users can modify the conversion parameters through a graphical interface.
[0129] To ensure the accuracy and efficiency of the conversion, the embodiments of the present invention also introduce the following optimization and verification mechanisms:
[0130] Data integrity verification: During the conversion process, CRC verification is performed on each data frame to ensure that the data frame has not had an error during transmission and storage.
[0131] Timestamp synchronization: Synchronize the timestamps between the FPGA and the host computer to ensure that the timestamps in the Pcap file accurately reflect the capture time of the frame.
[0132] The function of converting network data frames into the Pcap format recognizable by Wireshark enables users to perform detailed frame-level analysis using network analysis tools such as Wireshark. This conversion mechanism not only preserves the integrity and timing information of the data frames but also supports traffic playback, protocol parsing, and statistical analysis, greatly enhancing the flexibility and practicality of network traffic analysis.
[0133] Converting the Ethernet MAC frames captured by the FPGA into the Pcap format recognizable by Wireshark requires the following steps:
[0134] Extract data frames
[0135] Extract the Ethernet MAC frames from the raw data captured by the FPGA, including the destination MAC address, source MAC address, length / type field, data field, and FCS. Note that the FCS may be filtered out by the hardware in some scenarios, so it is necessary to decide whether to retain it according to the actual situation.
[0136] Generate the Pcap file header
[0137] At the beginning of the conversion, first generate the global file header of the Pcap file. The content of the file header includes: Magic number: The fixed value "0xA1B2C3D4", indicating the format of the Pcap file. Timestamp precision: Usually can be set to 0, indicating second-level precision. Maximum frame length: Can be set to 65535, indicating support for frames of the maximum length. Link type: Can be set to 1, indicating an Ethernet link.
[0138] Generate packet records
[0139] For each Ethernet MAC frame, generate the corresponding Pcap packet record, including the data packet header and the data frame: Data packet header: Timestamp: Extract the timestamp from the frame description header captured by the FPGA and convert it into the format of seconds and microseconds. Capture length: Set to the actual length of the frame (from the destination MAC address to the FCS). Original length: The same as the capture length. Data frame: Use the extracted Ethernet MAC frame directly as the data frame part of the packet record.
[0140] Write to the Pcap file
[0141] Write the generated Pcap file header and all packet records to the file in sequence to form a complete Pcap binary file. This file can be directly imported into Wireshark for traffic analysis and protocol parsing.
[0142] When the test is finished or the traffic is idle, the FPGA will upload the data frames stored in the DDR storage module to the host computer through the PCIE interface. During the reading process, the data integrity protection mechanism continues to work. When the FPGA reads the first data frame from the DDR storage module, it will recalculate the CRC checksum and compare it with the checksum attached to the end of the frame. If the check fails, it means that the data may have been damaged during storage or reading. The FPGA will record the error log and try to read it again. When uploading data to the host computer through the PCIE interface, the FPGA will use the error detection mechanism (such as ECRC) of the PCIE protocol to protect the data and ensure the integrity of the data during transmission.
[0143] It is understandable that most traditional conversion solutions perform format conversion on the host computer side, and conventional conversion tools use software parsing, which takes up a lot of CPU resources and hard disk space. In response to the above problems, the embodiment of the present invention adopts hardware-level Pcap encapsulation, realizes real-time generation of Pcap header in FPGA, contains nanosecond-level precision timestamp, and encapsulates it into an IP core for easy calling and parameter modification; the IP core configuration can support the generation of standard Pcap or ordinary Ethernet format; and adds a custom identifier (such as stream ID number) to the same data stream to enhance the analysis relevance.
[0144] PCIE reporting submodule
[0145] The PCIE reporting submodule uploads the reporting data packet to the host computer connected to the traffic analysis board based on the dynamic interrupt trigger mechanism of traffic idle detection, which may include:
[0146] After the test is completed or when the traffic is idle, the PCIE interface in the PCIE reporting submodule is used to send an interrupt signal to the host computer, and the reporting data packets that need to be uploaded are transmitted to the host computer at one time through the PCIE interface.
[0147] In high-traffic network test scenarios, the PCIE interface serves as a data transmission channel between the FPGA and the host computer, and its transmission efficiency directly affects the real-time nature of data reporting and the overall system performance. In order to complete the reporting of a large amount of data in a short time, the embodiment of the present invention optimizes the PCIE transmission protocol and combines it with efficient hardware design to achieve efficient data transmission of the PCIE interface.
[0148] Optimization of PCIE transmission protocol
[0149] Traditional PCIe data transmission usually adopts the polling method, that is, the host computer continuously queries whether the FPGA has data to be transmitted. Although this method is simple to implement, it will occupy a large amount of CPU resources in high-traffic scenarios, resulting in low transmission efficiency. To solve this problem, the present invention adopts the interrupt method to replace the polling method, and the specific optimization is as follows: Interrupt trigger mechanism: When there is data ready to be uploaded in the FPGA, an interrupt signal will be sent to the host computer through the PCIe interface to notify the host computer to read the data. This method avoids the overhead of continuous querying by the host computer and significantly improves the transmission efficiency. Batch data transmission: Each time an interrupt is triggered, the FPGA will pack multiple data frames into a batch data packet and transmit it to the host computer through the PCIe interface at one time. This method reduces the frequency of interrupt triggers and further improves the transmission efficiency. Transmission priority management: For high-priority data, the FPGA will give priority to triggering an interrupt and uploading it to ensure that critical data can be transmitted to the host computer in a timely manner.
[0150] Application of XDMA IP Core
[0151] To achieve efficient data transmission of the PCIe interface, the present invention adopts the XDMA (DMA-based PCIe Subsystem for PCI Express) IP core provided by Xilinx. The XDMA IP core is a high-performance DMA (Direct Memory Access) engine dedicated to data transmission of the PCIe interface. Its core advantages are as follows:
[0152] Hardware acceleration: The XDMA IP core realizes data transmission through hardware without the participation of the CPU, significantly reducing the resource occupancy at the host computer end.
[0153] High-bandwidth support: The XDMA IP core supports interfaces such as PCIe Gen3 x8 and even higher specifications, and can provide a theoretical bandwidth of up to 64 Gbps to meet the data transmission requirements in high-traffic scenarios.
[0154] Flexible configuration: The XDMA IP core supports multiple transmission modes (such as block transmission, stream transmission) and interrupt mechanisms, and can be flexibly configured according to actual needs.
[0155] The schematic diagram of the process for the PCIe reporting sub-module in the embodiment of the present invention to upload the reporting data packet is as Figure 5 shown. Specifically, at the FPGA end, the specific implementation of efficient PCIe data transmission includes the following key steps:
[0156] Data Packing: The FPGA packs the data frames stored in the DDR memory into data blocks of a fixed size and adds necessary metadata (such as frame description headers, CRC check codes, etc.).
[0157] DMA Transfer: Through the XDMA IP core, the FPGA directly transfers the packed data blocks to the memory of the host computer without CPU intervention. The XDMA IP core automatically manages the transfer process of the PCIE interface, including address mapping, data segmentation, and error retransmission, etc.
[0158] Interrupt Notification: When a batch of data transfer is completed, the XDMA IP core sends an interrupt signal to the host computer to notify the host computer to process the data. After receiving the interrupt, the host computer reads the data from the memory and performs subsequent analysis.
[0159] On the host computer side, the specific implementation of high - speed PCIE data transfer includes the following key steps:
[0160] Open the device file: The software first opens the device file related to the DMA device for communication with the hardware.
[0161] Configure DMA transfer: The software configures the parameters of the DMA transfer, such as source address, destination address, transfer size, etc. These parameters are usually passed to the driver through system calls such as ioctl or mmap.
[0162] Create a file name according to the current time and save it to the hard disk: The software generates a unique file name according to the current time for saving the data transferred by DMA.
[0163] Start DMA transfer listening: The software starts the DMA transfer and begins to listen for interrupt signals to detect whether the transfer is completed.
[0164] Check if an interrupt notification is received: The software checks whether it has received an interrupt notification of the completion of the DMA transfer through the interrupt handling mechanism.
[0165] Read the data and save it in a structure: If an interrupt notification is received, the software reads the data from the DMA buffer and saves it in a predefined structure.
[0166] Save the data to the hard disk: The software saves the read data to the file generated before and stores it on the hard disk.
[0167] The entire process is based on interrupt-driven DMA transfer to ensure the efficiency and real-time nature of data transfer. By optimizing the PCIE transfer protocol, adopting the XDMA IP core, and introducing an interrupt mechanism, the present invention achieves efficient data transfer through the PCIE interface. This design not only significantly improves the efficiency of data reporting but also ensures the real-time nature and integrity of data transfer, providing strong support for data storage and analysis in high-traffic network test scenarios.
[0168] It can be understood that traditional reporting mechanisms usually adopt polling or fixed-period interrupts, and interrupt handling is usually separated from data transfer. The embodiments of the present invention adopt an intelligent interrupt trigger strategy, a dynamic interrupt trigger mechanism based on traffic idle detection, and can freely set the traffic silent window threshold. For example, data transfer only starts when the port line speed is below 10%, avoiding excessive CPU resource occupation by the board; adopting time-division multi-channel DMA, dividing the PCIE channel into multiple virtual channels and multiplexing them by priority in a time-sharing manner; through data pre-packaging, data unit packaging is completed on the FPGA side to reduce the processing overhead of the host computer. By adopting the dynamic interrupt trigger mechanism based on traffic idle detection, the data transfer efficiency is significantly improved, ensuring the real-time nature of data reporting; at the same time, the data integrity mechanism (CRC check) ensures the accuracy of data during transmission and storage, avoiding data corruption caused by transmission errors or storage anomalies.
[0169] After the data is uploaded to the host computer, the software of the host computer will perform a final CRC check to ensure that no errors have occurred during the entire process of data transmission and storage. The specific steps are as follows:
[0170] The host computer software recalculates the CRC check code for each received data frame and compares it with the check code appended at the end of the frame. If the check passes, it indicates that the data is complete and error-free; if the check fails, error information is recorded and the corresponding processing logic is triggered (marked as corrupted data).
[0171] For data frames with failed checks, the software of the host computer will take different processing measures according to the error type and severity. For example, for non-critical data frames, they can be directly discarded; for critical data frames, they can be logged and the user can be notified for further analysis.
[0172] It can be understood that to further improve the reliability of the system, the embodiments of the present invention also design the functions of error logging and statistical analysis:
[0173] Error logging: At the host computer end, all data frames with failed checks will be recorded in the error log, including information such as frame number, error type, and timestamp, facilitating subsequent problem troubleshooting.
[0174] Statistical analysis: The software of the host computer will perform statistical analysis on the error logs, generate an error rate report, and help users understand the operating status of the system and the data integrity situation. For example, it can count the error rates in different time periods, analyze the distribution of error types, and evaluate the reliability of the DDR memory and PCIE interfaces.
[0175] In the embodiment of the present invention, through the combination of the FPGA and the DDR storage module, the large capacity and high bandwidth characteristics of the DDR storage module are utilized to enable the data frames in the network to be cached in real time, ensuring that all received data frames can be stored even at high traffic peaks and avoiding data loss; the dynamic partition storage method is used to store data units, improving the writing efficiency and reducing the possibility of packet loss. The present invention uses a dynamic interrupt trigger mechanism to replace the traditional polling method, significantly improving the data transmission efficiency and ensuring the real-time nature of data upload. Before the data is uploaded to the host computer, the data units are pre-converted into reported data packets in Pcap format. By pre-converting the data, not only the integrity and timing information of the data frames are retained, but also traffic playback, protocol parsing, and statistical analysis are supported, greatly enhancing the flexibility and practicality of network traffic analysis.
[0176] It should be noted that in the description of the present invention, it should be understood that the terms "first" and "second" are only used for descriptive purposes and cannot be construed as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, the features defined with "first" and "second" may explicitly or implicitly include one or more of such features. In the description of the present invention, "a plurality" means two or more unless otherwise specifically defined.
[0177] The above are only the preferred embodiments of the present invention and are not intended to limit the protection scope of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention are all included in the protection scope of the present invention.
Claims
1. A traffic analysis board based on DDR efficient storage, characterized in that: include: Power module, SFP optical port transceiver, clock module, FPGA and DDR storage module; among them, The power supply module is used to supply power to each module in the flow analysis board; The SFP optical port transceiver is used to receive data frames in the network; The clock module is used to provide a clock signal for the FPGA to work; The FPGA is used to pack the data frames received by the SFP optical port transceiver into data units; The DDR storage module is used to store the data unit based on a dynamic partition storage method under the control of the FPGA; The FPGA is also used to convert the data units stored in the DDR storage module into reporting data packets in Pcap format; and upload the reporting data packets to a host computer connected to the traffic analysis board based on a dynamic interrupt trigger mechanism for traffic idle detection.
2. According to the DDR efficient storage based traffic analysis board card of claim 1, it is characterized in that: The FPGA comprises: Input submodule, storage control submodule, format conversion submodule and PCIE reporting submodule; among which, The input submodule is used to pack the data frames received by the SFP optical port transceiver into data units; The storage control submodule is used to control the DDR storage module to store the data unit; The format conversion submodule is used to convert the data unit stored in the DDR storage module into a reporting data packet in Pcap format; The PCIE reporting submodule is used to upload the reporting data packet to a host computer connected to the traffic analysis board based on a dynamic interrupt trigger mechanism of traffic idle detection.
3. A traffic analysis board based on DDR high-efficiency storage according to claim 2, characterized in that: The input submodule packages the data frame into data units, including: The input submodule calculates the frame length of the data frame and records the receiving time point as a timestamp, and uses the frame length and the timestamp as a frame description header; Calculating a CRC check code on the data frame to obtain a CRC check code; The frame description header is added to the front of the data frame, and the CRC check code is added to the tail of the data frame to obtain the data unit, and the data unit is temporarily stored in its own buffer FIFO.
4. A traffic analysis board based on DDR high-efficiency storage according to claim 3, characterized in that: The DDR storage module stores the data unit based on a dynamic partition storage method under the control of the FPGA, including: The FPGA verifies the CRC check code in the data unit and sends the complete data unit to the DDR storage module; The AXI write address channel in the DDR storage module calculates the length of the burst transmission according to the frame length of the data frame and generates a continuous write address; According to the continuous write addresses, based on the dynamic partition storage method, the data units are stored in the corresponding positions of the DDR storage module.
5. A traffic analysis board based on DDR high-efficiency storage according to claim 4, characterized in that: The DDR storage module comprises: Data frame storage area, metadata storage area and management area; wherein, The data frame storage area is used to store the data frames in the data unit; The metadata storage area is used to store the frame description header, timestamp and CRC check code in the data unit; The management area is used to store the management information of the DDR memory.
6. The traffic analysis board based on DDR high-efficiency storage according to claim 2 is characterized in that: The format conversion submodule converts the data unit stored in the DDR storage module into a reporting data packet in Pcap format, including: The format conversion submodule is a preset IP core, and the format conversion submodule reads a complete data frame from a data unit stored in a DDR storage module; The data frame is encapsulated according to the Pcap format to obtain a reporting data packet in the Pcap format.
7. The traffic analysis board based on DDR high-efficiency storage according to claim 2 is characterized in that: The Pcap format of the reported data packet includes: File header, packet record, packet header and data frame.
8. The traffic analysis board based on DDR high-efficiency storage according to claim 2 is characterized in that: The PCIE reporting submodule uploads the reporting data packet to a host computer connected to the traffic analysis board based on a dynamic interrupt trigger mechanism of traffic idle detection, including: After the test is completed or when the traffic is idle, the PCIE interface in the PCIE reporting submodule is used to send an interrupt signal to the host computer, and the reporting data packets that need to be uploaded are transmitted to the host computer at one time through the PCIE interface.
Citation Information
Cited By
High-performance Netflow message output method and system and storage medium
CN120710947A
High-performance Netflow message output methods, systems, and storage media
CN120710947B
