Method and system for high-speed acquisition, analysis and storage of ship network data

Abstract

The invention discloses a ship network data-oriented high-speed acquisition, analysis and storage method and system, and the method comprises the following steps: 1) when a network data packet arrives at an FPGA (Field Programmable Gate Array), the FPGA verifies the network data packet; if the verification is not passed, discarding the message; (2) after the verification is passed, the FPGA filters the collected original data according to a configured filtering rule; 3) performing data content-level filtering on the filtered data according to a configured rule template, and storing the data in an FPGA DDR (Double Data Rate); and 4) the FPGA notifies application software to carry out data storage and carries the data to a storage module. According to the invention, matching filtering and content analysis preprocessing are realized through the FPGA according to a configuration filtering rule and a content format template rule before front-end data acquisition, then high-speed data acquisition is realized through the FPGA, and the acquired data are directly stored in the storage module, so that a data storage path does not pass through a CPU, and end-to-end high-performance storage of the data is realized.

Description

Technical Field

[0001] The present invention relates to data storage technology, and in particular to a method and system for high-speed collection, analysis and storage of ship network data. Background Art

[0002] There are two common solutions for existing network data collection and storage: 1. CPU-based real-time analysis and storage solution; 2. FPGA-based real-time analysis and storage solution;

[0003] The above two solutions have the following problems: 1) In the case of high-speed acquisition, analysis and storage based on the CPU real-time analysis and storage solution, data acquisition, analysis and storage operations are completely dependent on the CPU capabilities. If a certain link cannot be processed, data loss may occur. At the same time, when the CPU's ability to process data is weak, the real-time nature of the data cannot be guaranteed. The FPGA-based real-time analysis and storage solution solves the problem of high-speed acquisition at the front end, but the solution still needs to copy the data once, and does not solve the storage performance bottleneck. The performance is limited by the CPU capabilities, storage bodies, etc. 2) Both the CPU-based and FPGA-based solutions only support fixed standard five-tuple parsing and matching. For the FPGA acquisition mode, FPGA five-tuple parsing and matching can achieve partial data filtering, which improves efficiency to a certain extent. However, the data content analysis and processing are completely dependent on the application software and CPU capabilities, especially in high-speed (10G / 40G) linear acquisition, analysis and storage. If the processing is not timely, data loss will occur.

[0004] As the scale of business in real application scenarios expands, the number of Internet nodes in the network environment increases day by day, and the total data traffic also increases dramatically. In the face of the above changes, the use of traditional acquisition equipment will lead to delays in key data processing, thereby causing information system response lags and affecting the improvement of combat effectiveness. Therefore, there is an urgent need for a system deployed at the edge with high-performance acquisition, storage and analysis. Summary of the invention

[0005] The technical problem to be solved by the present invention is to provide a method and system for high-speed collection, analysis and storage of ship network data in view of the defects in the prior art.

[0006] The technical solution adopted by the present invention to solve the technical problem is: a method for high-speed collection, analysis and storage of ship network data, comprising the following steps:

[0007] 1) When a network data packet arrives at the FPGA, the FPGA verifies the network data packet; if the verification fails, the message is discarded;

[0008] 2) After verification, the FPGA filters the collected raw data according to the configured filtering rules according to the five-tuple (source IP, target IP, source port, target port and protocol type);

[0009] 3) The filtered data is filtered at the data content level according to the configured rule template;

[0010] After the quintuple is successfully matched, the FPGA parses and processes the data content according to the configuration data format template and stores the data in the FPGA DDR;

[0011] 4) FPGA real-time calculation: when the amount of data in the FPGA DDR reaches the threshold that can be transported, the application software is notified to store the data in an interrupt mode and transfer the data to the storage module;

[0012] The parameters sent by FPGA to the application software include: the first address of the current data in FPGA DDR, the valid length of the data, and the logical channel; the details are as follows:

[0013] 4.1) The application software monitors whether DMA interrupts are generated in real time. When receiving a DMA interrupt, the application software receives a data storage command. The application software first parses the parameters, then finds the corresponding file of the channel according to the logical channel ID, and finally calls the file system's extended interface to pass the FPGA DDR first address, valid length, and file handle to the file system;

[0014] 4.2) The file system allocates storage resources according to the file handle and effective length. At the same time, it needs to convert the CPU DDR physical address of the PCIe bus space into the FPGA DDR address, so that direct storage between the FPGA and the storage module can be realized without going through the CPU. Finally, the storage module driver submits an IO command to the storage module, and its parameters include the data storage FPGA DDR first address, effective length and storage logical address.

[0015] 4.3) The storage module receives the IO command, starts DMA, and moves the data in the FPGA DDR to the corresponding block address through the PCIe bus addressing method, realizing direct storage from FPGA to the storage module.

[0016] If it meets the display rules, it will be directly sent to the user display console for display. The storage module consists of multiple storage bodies.

[0017] A high-speed collection, analysis and storage system for ship network data, comprising:

[0018] In the acquisition module, the FPGA first determines whether the user has issued a filtering rule. If the filtering rule is not configured, the FPGA directly transfers the original data to the storage module through DMA. If the filtering rule is configured, the FPGA performs data analysis, and the parsed data is notified to the recording module for data storage;

[0019] Data analysis is as follows: The network data packet is checked through FPGA; if the check fails, the message is discarded;

[0020] After verification, the FPGA filters the collected raw data according to the configured filtering rules by five tuples (source IP, destination IP, source port, destination port, and protocol type);

[0021] The filtered data is filtered at the data content level according to the configured rule template;

[0022] After the five-tuple is successfully matched, the FPGA parses the data content according to the configuration data format template and stores the data in the FPGA DDR. When the amount of data in the FPGA DDR reaches the threshold that can be transported, the application software is notified to store the data and move the data to the storage module.

[0023] Recording module: Application software is deployed in the recording module. The application software monitors whether DMA interrupts are generated in real time. When receiving a DMA interrupt, it starts to parse parameters and find corresponding data according to the logical channel. After processing, the DMA process is started to move the data in FPGADDR directly to the storage body.

[0024] The acquisition module analyzes the data content after content-level filtering. If it meets the display rules, the data is directly sent to the user display console for display.

[0025] 1) The application software monitors whether DMA interrupts are generated in real time. When receiving a DMA interrupt, the application software receives a data storage command. The application software first parses the parameters, then finds the corresponding file of the channel according to the logical channel ID, and finally calls the file system's extended interface to pass the FPGA DDR first address, valid length, and file handle to the file system;

[0026] 2) The file system allocates storage resources according to the file handle and effective length. At the same time, it needs to convert the CPU DDR physical address of the PCIe bus space into the FPGA DDR address, so that direct storage between the FPGA and the storage module can be realized without going through the CPU. Finally, the storage module driver submits an IO command to the storage module, and its parameters include the data storage FPGA DDR first address, effective length and storage logical address.

[0027] 3) The storage module receives the IO command, starts DMA, and moves the data in the FPGA DDR to the corresponding block address through the PCIe bus addressing method, realizing direct storage from FPGA to the storage module.

[0028] Data exchange module, used for data transmission between the acquisition module, the recording module and the storage module. The data transmission adopts the standard PCIe protocol;

[0029] The storage module is used to receive the original data collected by the collection module.

[0030] The beneficial effects produced by the present invention are:

[0031] 1. The present invention uses FPGA to configure quintuple filtering rules and content format template rules before front-end data collection to implement quintuple-based matching filtering and content analysis preprocessing, and then uses FPGA to implement high-speed data collection and directly store the collected data in the storage module, thereby achieving a data storage path that does not pass through the CPU and realizing end-to-end high-performance data storage. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The present invention will be further described below with reference to the accompanying drawings and embodiments, in which:

[0033] Figure 1 is a method flow chart of an embodiment of the present invention;

[0034] Figure 2 It is a data principle diagram of an embodiment of the present invention. DETAILED DESCRIPTION

[0035] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0036] like Figure 1 and Figure 2 As shown, a method for high-speed collection, analysis and storage of ship network data includes the following steps:

[0037] 1) When a network data packet arrives at the FPGA, the FPGA verifies the network data packet; if the verification fails, the message is discarded;

[0038] 2) After verification, the FPGA filters the collected raw data according to the configured filtering rules according to the five-tuple (source IP, target IP, source port, target port and protocol type);

[0039] 3) The filtered data is filtered at the data content level according to the configured rule template;

[0040] After the quintuple is successfully matched, the FPGA parses and processes the data content according to the configuration data format template and stores the data in the FPGA DDR;

[0041] 4) FPGA real-time calculation: when the amount of data in the FPGA DDR reaches the threshold that can be transported, the application software is notified to store the data in an interrupt mode and transfer the data to the storage module;

[0042] The parameters sent by FPGA to the application software include: the first address of the current data in FPGA DDR, the valid length of the data, and the logical channel; the details are as follows:

[0043] 4.1) The application software monitors whether DMA interrupts are generated in real time. When receiving a DMA interrupt, the application software receives a data storage command. The application software first parses the parameters, then finds the corresponding file of the channel according to the logical channel ID, and finally calls the file system's extended interface to pass the FPGA DDR first address, valid length, and file handle to the file system;

[0044] 4.2) The file system allocates storage resources according to the file handle and effective length. At the same time, it needs to convert the CPU DDR physical address of the PCIe bus space into the FPGA DDR address, so that direct storage between the FPGA and the storage module can be realized without going through the CPU. Finally, the storage module driver submits an IO command to the storage module, and its parameters include the data storage FPGA DDR first address, effective length and storage logical address.

[0045] 4.3) The storage module receives the IO command, starts DMA, and moves the data in the FPGA DDR to the corresponding block address through the PCIe bus addressing method, realizing direct storage from FPGA to the storage module.

[0046] After parsing the data content after content-level filtering, if it meets the display rules, the data is directly sent to the user display console for display. The storage module consists of multiple storage bodies.

[0047] The present invention also provides a high-speed collection, analysis and storage system for ship network data, comprising:

[0048] In the acquisition module, the FPGA first determines whether the user has issued a filtering rule. If the filtering rule is not configured, the FPGA directly transfers the original data to the storage module through DMA. If the filtering rule is configured, the FPGA performs data analysis, and the parsed data is notified to the recording module for data storage;

[0049] The acquisition module is used for user network data acquisition in the core network switch: FPGA can decide whether to filter the original data according to the actual needs of the user. If the user does not configure the filtering rules, FPGA will cache the collected original data in DDR first. When the cached data volume reaches the threshold or reaches the set timeout time, it will notify the application software to move the data. After receiving the data movement instruction, the application software completes the command parsing and moves the data in FPGADDR to the storage module; if the user configures the filtering rules, FPGA will filter the collected original data according to the configured filtering rules according to the five-tuple. The filtered data can be filtered at the data content level according to the rule template, and then the final result data will be sent to the recording module application software. After receiving the data, the application software on the recording module can directly send it to the user display console for direct display, and move the data in FPGADDR to the storage module;

[0050] Data analysis is as follows: The network data packet is checked through FPGA; if the check fails, the message is discarded;

[0051] After verification, the FPGA filters the collected raw data according to the configured filtering rules by five tuples (source IP, destination IP, source port, destination port, and protocol type);

[0052] The filtered data is filtered at the data content level according to the configured rule template;

[0053] After the five-tuple is successfully matched, the FPGA parses and processes the data content according to the configuration data format template, and stores the data in the FPGA DDR. When the amount of data in the FPGA DDR reaches the threshold that can be transported, the application software is notified to store the data and move the data to the storage module.

[0054] Recording module: Application software is deployed in the recording module. The application software monitors whether DMA interrupts are generated in real time. When receiving a DMA interrupt, it starts to parse parameters and find the corresponding data according to the logical channel. After processing, the DMA process is started to move the data in the FPGA DDR directly to the storage body.

[0055] The acquisition module analyzes the data content after content-level filtering. If it meets the display rules, the data is directly sent to the user display console for display.

[0056] 1) The application software monitors whether DMA interrupts are generated in real time. When receiving a DMA interrupt, the application software receives a data storage command. The application software first parses the parameters, then finds the corresponding file of the channel according to the logical channel ID, and finally calls the file system's extended interface to pass the FPGA DDR first address, valid length, and file handle to the file system;

[0057] 2) The file system allocates storage resources according to the file handle and effective length. At the same time, it needs to convert the CPU DDR physical address of the PCIe bus space into the FPGA DDR address, so that direct storage between the FPGA and the storage module can be realized without going through the CPU. Finally, the storage module driver submits an IO command to the storage module, and its parameters include the data storage FPGA DDR first address, effective length and storage logical address.

[0058] 3) The storage module receives the IO command, starts DMA, and moves the data in the FPGA DDR to the corresponding block address through the PCIe bus addressing method, realizing direct storage from FPGA to the storage module.

[0059] Data exchange module, used for data transmission between the acquisition module, the recording module and the storage module. The data transmission adopts the standard PCIe protocol;

[0060] A storage module, used for receiving the raw data collected by the collection module;

[0061] The raw data collected by the acquisition module changes the data stream transmission path through the application software, and is directly written into the storage module through direct memory access DMA to achieve high-performance data storage.

[0062] The reason why the present invention realizes the change of data storage path is that the file system transmits the data storage address to the driver, which is changed from the CPU DDR address to the FPGA DDR address, and then the data is directly sent from the FPGA DDR to the storage module through the PCIE Switch bus addressing, thereby realizing data storage independent of CPU performance and realizing high-performance storage of network data.

[0063] It can be seen that the method of the present invention can break through the CPU performance bottleneck. In the case of single-channel 10Gb optical fiber transmission, the acquisition and storage performance reaches 1000MB / s, which is much greater than the performance limit of CPU acquisition and storage. At the same time, the data can be stamped with microsecond timestamps through FPGA during data storage for subsequent data real replay. It is suitable for the acquisition, analysis and storage of network data in various types of ship network systems.

[0064] It should be understood that those skilled in the art can make improvements or changes based on the above description, and all these improvements and changes should fall within the scope of protection of the appended claims of the present invention.

Claims

1. A method for high-speed collection, analysis and storage of ship network data, characterized in that: The following steps are involved: 1) When a network data packet arrives at the FPGA, the FPGA verifies the network data packet; if the verification fails, the message is discarded; 2) After verification, the FPGA filters the collected raw data according to the configured filtering rules; 3) The filtered data is filtered at the data content level according to the configured rule template, and the data is stored in the FPGA DDR; Among them, data content level filtering is that FPGA parses and processes the data content according to the configuration data format template; 4) FPGA notifies the application software to store data and move the data to the storage module; The parameters sent by the FPGA to the application software include: the first address of the current data in the FPGA DDR, the valid length of the data, and the logic channel.

2. The method for high-speed collection, analysis and storage of ship network data according to claim 1 is characterized in that: In the step 2), the FPGA filters the collected raw data according to the configured filtering rules according to the five-tuple (source IP, target IP, source port, target port and protocol type).

3. The method for high-speed collection, analysis and storage of ship network data according to claim 1 is characterized in that: In step 4), the FPGA performs real-time calculations, and when the amount of data in the FPGA DDR reaches a transportable threshold, the FPGA notifies the application software to store the data.

4. The method for high-speed collection, analysis and storage of ship network data according to claim 1 is characterized in that: In the step 4), the FPGA notifies the application software to store data in an interrupt manner.

5. The method for high-speed collection, analysis and storage of ship network data according to claim 4 is characterized in that: In step 4), the FPGA notifies the application software to perform data storage in an interrupt manner, as follows: 4.1) The application software monitors whether DMA interrupts are generated in real time. When receiving a DMA interrupt, the application software receives a data storage command. The application software first parses the parameters, then finds the corresponding file of the channel according to the logical channel ID, and finally calls the file system's extended interface to pass the FPGA DDR first address, valid length, and file handle to the file system; 4.2) The file system allocates storage resources according to the file handle and effective length. At the same time, it needs to convert the CPU DDR physical address of the PCIe bus space into the FPGA DDR address, so that direct storage between the FPGA and the storage module can be realized without going through the CPU. Finally, the storage module driver submits an IO command to the storage module, and its parameters include the data storage FPGA DDR first address, effective length and storage logical address. 4.3) The storage module receives the IO command, starts DMA, and moves the data in the FPGA DDR to the corresponding block address through the PCIe bus addressing method, realizing direct storage from FPGA to the storage module.

6. The method for high-speed collection, analysis and storage of ship network data according to claim 1 is characterized in that: In the step 3), after the data is filtered at the content level, if it meets the display rules, the data is directly sent to the user display console for display.

7. The method for high-speed collection, analysis and storage of ship network data according to claim 1 is characterized in that: In the step 4), the storage module is composed of a plurality of storage bodies.

8. A high-speed collection, analysis and storage system for ship network data, characterized in that: include: In the acquisition module, the FPGA first determines whether the user has issued a filtering rule. If the filtering rule is not configured, the FPGA directly transfers the original data to the storage module through DMA. If the filtering rule is configured, the FPGA performs data analysis, and the parsed data is notified to the recording module for data storage; Data analysis is as follows: The network data packet is checked through FPGA; if the check fails, the message is discarded; After verification, the FPGA filters the collected raw data according to the configured filtering rules by five tuples (source IP, destination IP, source port, destination port, and protocol type); The filtered data is filtered at the data content level according to the configured rule template; After the quintuple is successfully matched, the FPGA parses and processes the data content according to the configuration data format template and stores the data in the FPGA DDR; When the amount of data in the FPGA DDR reaches the threshold that can be transported, the application software is notified to store the data and move the data to the storage module; Recording module: Application software is deployed in the recording module. The application software monitors whether DMA interrupts are generated in real time. When receiving a DMA interrupt, it starts to parse the parameters and find the corresponding data according to the logical channel. After processing, the DMA process is started to move the data in the FPGA DDR directly to the storage body; the details are as follows: 1) The application software monitors whether DMA interrupts are generated in real time. When receiving a DMA interrupt, the application software receives a data storage command. The application software first parses the parameters, then finds the corresponding file of the channel according to the logical channel ID, and finally calls the file system's extended interface to pass the FPGA DDR first address, valid length, and file handle to the file system; 2) The file system allocates storage resources according to the file handle and effective length. At the same time, it needs to convert the CPUDDR physical address of the PCIe bus space into the FPGA DDR address, so that direct storage between the FPGA and the storage module can be realized without going through the CPU. Finally, the storage module driver submits an IO command to the storage module, and its parameters include the data storage FPGA DDR first address, effective length and storage logical address. 3) The storage module receives the IO command, starts DMA, and moves the data in the FPGA DDR to the corresponding block address through the PCIe bus addressing method, realizing direct storage from FPGA to the storage module. Data exchange module, used for data transmission between the acquisition module, the recording module and the storage module. The data transmission adopts the standard PCIe protocol; The storage module is used to receive the original data collected by the collection module.

9. An electronic device, characterized in that: include: one or more processors; as well as a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors are caused to execute the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images