A DPDK-based HTTP text restoration method

Abstract

The application discloses a kind of HTTP text restoration methods based on DPDK, it is related to computer network technical field, the method uses multi-process decoupling architecture, and traffic acquisition and protocol analysis, fragmentation reorganization and text restoration, formatted message splicing, text restoration timeout and the like process are respectively operated in independent CPU core, efficient, lock-free data transmission is carried out using the ring mechanism of DPDK.Different from traditional memory splicing scheme, the application calculates data offset based on TCP sequence number, and directly writes the load content into the corresponding position of the file, effectively solves the problems of message out-of-order, packet loss and retransmission, etc., adopts the atomic renaming mechanism from fileID1 (intermediate state) to fileID2 (completed state), to ensure the accurate migration of file state.In addition, the application innovatively proposes a bidirectional asynchronous scheduling mechanism based on data export and text restoration state monitoring, which enables the data export process to maintain high throughput continuously and ensures the overall processing capacity of the system.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer networks, and in particular to a HTTP text restoration method based on DPDK. BACKGROUND

[0002] HTTP (Hypertext Transfer Protocol) is the most widely used application layer protocol on the Internet, supporting the communication of most Web services, API interfaces, cloud storage, mobile Apps, etc. In addition, with the popularity of HTTPS (HTTP over TLS), although the number of clear HTTP has decreased, in many border detection devices, further data restoration and behavior analysis of HTTP content after TLS decryption are still required. In a real network environment, due to MTU limitations, unstable links, and large application layer packets, TCP packets are often split into multiple fragments for transmission, forming so-called TCP fragmentation, out-of-order, and retransmission packets. Therefore, it is difficult to accurately obtain complete HTTP packet content using only packet capture tools or protocol stack logs.

[0003] Traditional methods based on kernel protocol stacks or single-thread user mode splicing cannot stably process complex TCP fragmentation data in high-concurrency, bypass monitoring, or low-latency scenarios. Existing technologies use memory-based packet buffering methods for recombination, which have problems such as being susceptible to memory limitations, invalidity being unrecoverable, and high thread coupling. In the process of fragmentation recombination and subsequent HTTP content analysis, system maintainability and scalability are severely restricted.

[0004] For the above problems, the existing technology and the comparative patent analysis are as follows:

[0005] A kind of IP packet fragmentation recombination method and system based on FPGA in Chinese patent publication No. CN119420707A, a process of IP packet fragmentation recombination is unloaded from CPU to FPGA execution, solves the problem of high CPU load of IP fragmentation packet, and TCAM is used for IP packet fragmentation matching, avoiding the hash value conflict problem caused by the common matching method based on hash operation, but its processing logic is limited to hardware, and the defect of poor flexibility cannot be implemented in pure software environment, the application range is limited, and the processing still stays in IP layer, which is not suitable for complex HTTP application layer stream restoration processing;

[0006] The DPDK packet fragmentation processing method and device of Chinese patent publication No. CN116634044A proposes a DPDK-based packet fragmentation processing method, which judges whether there is packet information in the preset hash table, and if not, stores the fragmented packet in the hash table based on the packet information, and transfers the fragmented packet to the corresponding CPU for processing by extracting the corresponding logical core number. However, the scheme is insufficient in handling packet disorder, packet loss, retransmission and other abnormal conditions, and still has certain coupling, and fails to completely decouple the operations of packet fragmentation and reassembly and main process, lacking more fine-grained fault tolerance mechanism and robustness.

[0007] The DPDK-based high-speed network packet processing system and method of Chinese patent publication No. CN116996444A proposes a DPDK-based high-speed network packet processing system and method, which has the core advantage of improving the efficiency of packet processing by bypassing the operating system kernel, and is suitable for high-throughput network scenarios. However, its limitation lies in the lack of in-depth design for the reassembly process of TCP fragments, and the lack of processing of conditions such as TCP disorder, retransmission and packet loss in data packets. This scheme focuses more on system optimization at the framework level, and is insufficient in detail processing such as fragmented packet reassembly and text restoration, and cannot meet the high reliability requirements in complex network environments. SUMMARY

[0008] The present application aims to provide a DPDK-based HTTP text restoration method to solve the problems raised in the background art.

[0009] To solve the above technical problems, the technical solution adopted by the present application is:

[0010] A DPDK-based HTTP text restoration method, comprising the following steps:

[0011] Step 1, build a multi-process architecture based on DPDK, run process, reassembly, sink and monitor processes respectively, and each process performs asynchronous data transmission through DPDK ring;

[0012] Step 2, collect TCP fragmented HTTP stream through the process, parse and identify the connection meeting the conditions, and generate a unique identifier fileID combined with the five-tuple timestamp and core ID;

[0013] Step 3, name fileID as fileID1 and point to the temporary path, extract payload and TCP sequence number, and send to the reassembly process for reassembly processing through ring;

[0014] Step 4, the reassembly process calculates the offset position of the payload in the target file according to the TCP sequence number, writes the fileID1 file, supports fault tolerance recombination of out-of-order, retransmission and packet loss;

[0015] Step 5, when the recombination completion condition is met, the file pointed by fileID1 is atomically renamed as fileID2, and is moved to the success path, ensuring accurate migration of the file state;

[0016] Step 6, the sink process judges whether the file exists and reads the text content splicing result data according to the fileID2 carried by the analysis result, and if the file does not exist, the data is transmitted to the monitor process through the ring for asynchronous polling detection;

[0017] Step 7, the monitor process asynchronously polls the existence state of fileID2, notifies the sink process after completing the recombination, otherwise marks the recombination failure after timeout, the sink process reads the file content according to fileID2, splices the formatted data, and sends the formatted data to the middleware for use by the upper module.

[0018] The further improvement of the technical scheme of the application is that the step 1 specifically comprises:

[0019] The system environment is initialized by using the DPDK framework, including configuring a memory pool, creating a CPU affinity setting, initializing a ring structure of the DPDK for inter-process communication, and creating four independent processes, namely a process, a reassembly process, a sink process and a monitor process, each process is bound to a different CPU core to run, so as to realize decoupling and parallel processing between processes, wherein the process is responsible for traffic collection and protocol analysis, the reassembly process is responsible for fragmentation recombination and text restoration, the sink process is responsible for formatted message splicing, and the monitor process is responsible for text restoration timeout waiting;

[0020] According to the designed architecture, the processes are deployed to different CPU cores, the DPDK ring structure is created for each process, and the processes are asynchronously transmitted through the DPDK ring, forming a collaborative workflow.

[0021] The further improvement of the technical scheme of the application is that the step 2 specifically comprises:

[0022] The process obtains the TCP slice HTTP stream in the network through various ways including network card capture, pcap file reading or socket communication, and performs protocol analysis on the collected TCP slice, and identifies the message conforming to the HTTP protocol characteristics, including checking the header information of the TCP message, identifying the HTTP request and response message, etc.

[0023] In the analysis process, the process identifies each HTTP connection by analyzing the five-tuple of the TCP message, wherein each HTTP connection has a unique five-tuple identifier, and the five-tuple is the source IP address, the destination IP address, the source port, the destination port and the protocol type;

[0024] For each identified HTTP connection, a unique identifier, i.e., fileID, is generated by the process, which is composed of the five-tuple, the timestamp and the core ID of the CPU currently running the process, and the generated fileID is recorded, and the TCP slice message related to the HTTP connection is associated with the fileID and sent to the reassembly process through the DPDK ring structure for further processing.

[0025] The further improvement of the technical scheme of the application is that the fileID includes fileID1 and fileID2 in two states, and the generation method of the fileID includes the UUID generated based on the TCP five-tuple, the timestamp and the core ID of the process running, to ensure that the reassembled file of each session is unique and does not conflict.

[0026] The fileID1 is used to represent the intermediate file that has not been completed reassembled, and is located in the temporary path.

[0027] The fileID2 represents the file that has been completed reassembled or only partially completed reassembled, and is located in the success path, and the fileID1 and the fileID2 are converted by renaming to avoid misprocessing of the intermediate state file.

[0028] The further improvement of the technical scheme of the application is that the step 3 specifically includes:

[0029] The original data packet is captured by DPDK, the TCP header information is analyzed, the five-tuple is extracted, and if the data comes from a pcap file or a socket, the five-tuple also needs to be analyzed, a hash table is used to maintain the HTTP connection state, if the five-tuple already exists, the last active time is updated, if it does not exist, a new connection context is created, the first identification timestamp and the current CPU core ID are recorded, and then the fileID1 is generated based on the five-tuple, the timestamp and the core ID.

[0030] Check if the TCP load contains HTTP features, filter non-HTTP traffic, and extract key fields including payload, TCP sequence number and fileID1 from the message, where payload is the HTTP header and body data, TCP sequence number is used for out-of-order rearrangement, and fileID1 is used to associate the current connection, and then write the payload and sequence number to a temporary file, while binding the metadata in memory, if the message is fragmented, append it to the temporary file in sequence number order;

[0031] Construct a DPDK Ring message structure, including fileID1, temporary file path and sequence number range, and send the message structure to the reassembly process through the DPDK Ring, if the Ring is full, use spin waiting or back pressure mechanism to avoid packet loss.

[0032] The further improvement of the technical scheme of the application is that the step 4 specifically includes:

[0033] The reassembly process receives the message structure from the process process from the DPDK ring queue, which contains key information such as fileID1, temporary file path and sequence number range, and parses the message structure to extract fileID1 and related data, locate to the corresponding temporary file path, and then check if the corresponding temporary file exists according to fileID1, if the file does not exist, create a new temporary file according to fileID1, if the file already exists, prepare to write the file;

[0034] According to the TCP sequence number, calculate the offset position of the payload in the target file, wherein the TCP sequence number and the initial sequence number of the current message are obtained from the message structure, the offset is calculated, and the payload is written to the corresponding position of the fileID1 file according to the calculated offset, if it is the first packet, offset=0, write the payload at the beginning of the file, if it is a non-first packet, write the payload directly to the specified position of the file according to the offset.

[0035] The further improvement of the technical scheme of the application is that the reassembly process calculates the offset offset according to the difference between the TCP sequence number and the initial sequence number, that is, the offset position is written, and the fileID1 corresponding file is directly written, supporting fault tolerance recombination of out-of-order, retransmission and packet loss;

[0036] Wherein, the expression of the offset is:

[0037] ;

[0038] In the formula, is the offset of the payload of the current packet in the target file, used to determine which position in the file the data of the current packet should be written to, is the TCP sequence number of the current packet, is the initial sequence number, indicating the TCP sequence number of the first packet of the HTTP connection.

[0039] The further improvement of the technical scheme of the present application is that the step 5 specifically comprises:

[0040] In the reassembly process, after completing the payload write operation of the current packet, whether the reassembly completion condition is met is checked by comparing whether the current length of the file is consistent with the expected content-length. If the file length is equal to the content-length, it means that all data has been completely written, the reassembly is completed. If the file length is less than the content-length, it means that there are subsequent fragment packets that have not arrived, and the waiting continues. If the reassembly is not completed within the preset timeout, it is considered that the reassembly is completed;

[0041] When the reassembly is completed or timed out, an atomic renaming operation is performed to atomically rename the temporary file pointed to by the fileID1 to the fileID2. The renaming operation is completed by using the atomic file operation function provided by the operating system. After renaming, the file path is migrated from the temporary path to the success path.

[0042] After completing the atomic renaming, the file path is migrated from the temporary path to the success path, and the storage location of the file is updated, so that the reassembled file can be correctly recognized and processed by the downstream module. In the success path, the file is identified by the fileID2, indicating that the file has completed the reassembly, and is used for subsequent text restoration and analysis.

[0043] The further improvement of the technical scheme of the present application is that the step 6 specifically comprises:

[0044] The sink process receives data from the process in the DPDK ring queue, parses the data, extracts key information including the fileID2 and related metadata, and locates to the corresponding file path according to the fileID2, and prepares for the next operation.

[0045] According to the fileID2 in the analysis result, it is checked whether the file exists, if the file exists, the file content is read, the recombined HTTP text data is obtained, the text content is spliced with the fileID2 to form the formatted data, and the formatted data is sent to the middleware or the upper module, if the file does not exist, it is checked whether the current time and the data recording time exceed the preset waiting time threshold, if the threshold is not exceeded, the data is sent to the monitor process through the DPDK ring to request to perform the asynchronous polling detection, if the threshold is exceeded, the recombination failure is marked, the failure information is spliced to form the formatted data, and the formatted data is sent to the middleware or the upper module;

[0046] According to the file existence and the recombination state, the processing result data is processed, if the file exists and the recombination succeeds, the formatted data is sent to the middleware or the upper module, and the final output of the data is completed, if the file does not exist and the asynchronous polling detection is needed, the data is sent to the monitor process, and subsequent processing results are continued to be waited, if the recombination fails, the failure information is sent to the middleware or the upper module.

[0047] The further improvement of the technical scheme of the application is that the step 7 specifically comprises:

[0048] The monitor process receives the data from the sink process in the DPDK ring queue, parses the data to obtain the fileID2 and related metadata, positions to the corresponding file path according to the fileID2, records the receiving time of the data, and simultaneously starts the asynchronous polling mechanism to regularly check whether the fileID2 file exists, and the polling interval can be adjusted according to the system load and performance requirements.

[0049] In the polling process, the existence state of the fileID2 file is checked, if the file exists, the recombination success is marked, the success state information and the fileID2 are packaged, and are sent back to the sink process through the DPDK ring to inform that the recombination is completed, if the file does not exist, it is checked whether the current time and the data receiving time exceed the preset timeout threshold, if the threshold is not exceeded, the file is continued to be polled and waited, if the threshold is exceeded, the recombination failure is marked, the failure state information and the fileID2 are packaged, and are sent back to the sink process through the DPDK ring to inform that the recombination fails.

[0050] According to the polling result, the state information is sent back to the sink process, for the state information of the recombination success, after the sink process receives the success state, the file content is read according to the fileID2, the file content and the fileID2 are spliced to form the formatted data, and the formatted data is sent to the middleware for the use of the upper module, for the state information of the recombination failure, after the sink process receives the failure state, the failure information is spliced to form the formatted data, and the formatted data is sent to the middleware for the subsequent error processing or retry operation of the upper module.

[0051] Due to the adoption of the above technical solutions, the technical progress achieved by the present application relative to the prior art is:

[0052] The present application provides a HTTP text restoration method based on DPDK, which adopts a DPDK multi-process architecture, each function (such as fragmentation recombination, HTTP text restoration, data processing, etc.) independently runs on different CPU cores, and the processes communicate asynchronously through a Ring queue, having natural isolation and error tolerance capability, which can effectively reduce the system crash domain and improve the debuggability and expansibility of the system.

[0053] The present application provides a HTTP text restoration method based on DPDK, the prior art usually stores fragmentation message recombination information and cache in memory, which is easy to cause information loss under abnormal restart, memory leakage or large flow, and has high memory consumption, which is not conducive to engineering implementation, the present application stores the recombination content by using a file writing method, calculates the writing offset according to the TCP sequence number, accurately processes complex TCP scenarios such as out-of-order, packet loss and retransmission, and does not affect data recombination in the process of machine restart, having strong fault tolerance and replay capability.

[0054] The present application provides a HTTP text restoration method based on DPDK, which adopts an atomic renaming mechanism of fileID1 to fileID2, ensures clear state and process decoupling, improves the consistency and concurrent safety of the processing flow, and is easier to be consumed by downstream modules.

[0055] The present application provides a HTTP text restoration method based on DPDK, which temporarily transfers unfinished tasks to monitor by using a bidirectional asynchronous scheduling mechanism of sink and monitor, and sink keeps high throughput, which can greatly avoid the main process being slowed down by IO, and guarantees the overall system processing capability. BRIEF DESCRIPTION OF DRAWINGS

[0056] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiments will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments described in the present application, and other drawings can also be obtained by those skilled in the art according to these drawings.

[0057] Figure 1 The present application provides a HTTP text restoration method based on DPDK, which adopts a DPDK multi-process architecture, each function (such as fragmentation recombination, HTTP text restoration, data processing, etc.) independently runs on different CPU cores, and the processes communicate asynchronously through a Ring queue, having natural isolation and error tolerance capability, which can effectively reduce the system crash domain and improve the debuggability and expansibility of the system.

[0058] Figure 2 The present application provides a HTTP text restoration method based on DPDK, which adopts a DPDK multi-process architecture, each function (such as fragmentation recombination, HTTP text restoration, data processing, etc.) independently runs on different CPU cores, and the processes communicate asynchronously through a Ring queue, having natural isolation and error tolerance capability, which can effectively reduce the system crash domain and improve the debuggability and expansibility of the system. DETAILED DESCRIPTION

[0059] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0060] Example 1, as Figure 1 , Figure 2 As shown, this invention provides an HTTP text restoration method based on DPDK, comprising the following steps:

[0061] Step 1: Construct a multi-process architecture based on DPDK, running the process, reassembly, sink, and monitor processes. Each process asynchronously transfers data via a DPDK ring. Initialize the system environment using the DPDK framework, including configuring the memory pool, creating CPU affinity settings, and initializing the DPDK ring structure for inter-process communication. Create four independent processes: process, reassembly, sink, and monitor. Each process is bound to a different CPU core to achieve decoupling and parallel processing. The process is responsible for traffic collection and protocol parsing, the reassembly process for fragment reassembly and text reconstruction, the sink process for formatted message assembly, and the monitor process for text reconstruction timeout waiting. Based on the designed architecture, deploy each process to different CPU cores, creating a DPDK ring structure for each process. Each process communicates via DPDK. The ring enables asynchronous data transfer, forming a collaborative workflow. The process acquires the HTTP stream, parses key information, and sends it to the reassembly process via the ring. The reassembly process completes fragment reassembly and text reconstruction, then passes the results to the sink process. If the sink process finds that text reconstruction is incomplete while assembling the message, it passes the data to the monitor process via the ring for timeout detection. Based on the detection results, the monitor process feeds back the status information to the sink process, completing the final data assembly and output. Throughout this process, each process achieves efficient, low-latency data exchange through the DPDK ring, ensuring high throughput and low coupling of the system.

[0062] Step 2, the process process collects TCP fragment HTTP stream, parses and identifies the connection meeting the condition, generates a unique identifier fileID combined with the quintuple timestamp and core ID, the process process obtains the TCP fragment HTTP stream in the network through various ways including network card capture, pcap file reading or socket communication, and performs protocol analysis on the collected TCP fragments, identifies the messages meeting the HTTP protocol characteristics, including checking the header information of TCP message, identifying HTTP request and response message, etc. In the analysis process, the process process identifies each HTTP connection by analyzing the quintuple of TCP message, wherein each HTTP connection has its unique quintuple identifier, and the quintuple is the source IP address, destination IP address, source port, destination port and protocol type. For each identified HTTP connection, a unique identifier fileID is generated by the process process, which is composed of quintuple, timestamp and core ID (coreID) of the process process currently running, wherein the quintuple ensures the uniqueness of the connection at the network layer, the timestamp records the time when the connection is identified, which helps to distinguish different connection instances under the same quintuple in subsequent processing, and the core ID ensures that the process process instances running on different CPU cores can generate non-conflicting fileIDs, and then records the generated fileID, and associates the TCP fragment messages (including the payload and TCP sequence number) related to the HTTP connection with the fileID, and sends it to the reassembly process through the DPDK ring structure for further processing;

[0063] In addition, the fileID includes two states of fileID1 and fileID2, and the generation method of the fileID includes the UUID generated based on the TCP quintuple, timestamp and core ID of the process running, which ensures the uniqueness of the reassembly file of each session without conflict, wherein fileID1 is used to represent the intermediate file which has not completed reassembly, and is located in the temporary path, fileID2 represents the file which has completed reassembly or only partially completed reassembly, and is located in the success path, fileID1 and fileID2 realize state conversion through renaming, avoiding the intermediate state file being misprocessed;

[0064] Step 3, name fileID as fileID1 and point to the temporary path, extract payload and TCP sequence number, send to reassembly process through ring for reassembly, capture raw packets through DPDK, parse TCP header information, extract five-tuple, if data comes from pcap file or socket, also need to parse five-tuple, use hash table to maintain HTTP connection state, if five-tuple exists, update its last active time, if not, create new connection context, record first identification timestamp and current CPU core ID, then generate fileID1 based on five-tuple, timestamp and core ID, fileID1 is used to identify unique connection instance and point to temporary storage path, subsequent packets will be reassembled according to this path, check if TCP payload contains HTTP features, filter non-HTTP traffic, and extract key fields including payload, TCP sequence number and fileID1 from packets, where payload is HTTP header and body data, TCP sequence number is used for out-of-order rearrangement, and fileID1 is used to associate the current connection, then write payload and sequence number to temporary file (path pointed by fileID1), and bind metadata in memory at the same time, if the packet is fragmented, append to the temporary file in sequence number order, construct DPDK ring message structure, including fileID1, temporary file path and sequence number range, fileID1 is used to identify target connection, temporary file path is used to point to payload storage location, and sequence number range is used for reassembly verification, send message structure to reassembly process through DPDK ring, if ring is full, use spin waiting or back pressure mechanism to avoid packet loss;

[0065] Step 4, the reassembly process calculates the offset position of the payload in the target file according to the TCP sequence number, writes the fileID1 file, supports fault-tolerant recombination of out-of-order, retransmission and packet loss, the reassembly process receives a message structure from the process process in the DPDK ring queue, the structure contains the key information of fileID1, temporary file path and sequence number range, parses the message structure, extracts fileID1 and related data, locates to the corresponding temporary file path, and then checks whether the corresponding temporary file exists according to fileID1, if the file does not exist, creates a new temporary file according to fileID1, if the file already exists, prepares to write the file, calculates the offset position of the payload in the target file according to the TCP sequence number, wherein the TCP sequence number and the initial sequence number of the current message are obtained from the message structure, the offset is calculated, and the payload is written to the corresponding position of the fileID1 file according to the calculated offset, if it is the first packet, offset = 0, the payload is written at the beginning of the file, if it is not the first packet, the payload is directly written to the specified position of the file according to the offset, instead of simply appending the content, which can effectively handle the problems of out-of-order, retransmission and packet loss;

[0066] In addition, the reassembly process calculates the offset offset according to the difference between the TCP sequence number and the initial sequence number, that is, the writing offset position, and directly writes the fileID1 corresponding file, supports fault-tolerant recombination of out-of-order, retransmission and packet loss;

[0067] Wherein, the expression of the offset is:

[0068] ;

[0069] In the formula, is the offset, which indicates the offset position of the payload of the current message in the target file, and is used to determine where the data of the current message should be written in the file, is the TCP sequence number, which indicates the TCP sequence number of the current message, is a field extracted from the TCP message header, and the TCP sequence number is a mechanism used by TCP protocol to identify the order of data segments. Each TCP message segment has a sequence number, which is used to ensure reliable transmission of data. In the recombination process, the current sequence number is used to determine the position of the current message in the entire data stream, is the initial sequence number, which indicates the TCP sequence number of the first message of the HTTP connection, and is recorded when the connection is established. The initial sequence number is used as a reference point to calculate the offset of subsequent messages. By taking the difference between the current sequence number and the initial sequence number as the offset, the position of the current message in the entire file can be determined;

[0070] Step 5, when the reassembly completion condition is met, the file pointed to by fileID1 is atomically renamed to fileID2 and moved to the success path, ensuring accurate migration of file state. In the reassembly process, after completing the payload write operation on the current packet, the reassembly completion condition is checked by comparing the current file length with the expected content-length. If the file length equals the content-length, it means that all data has been completely written and reassembly is complete. If the file length is less than the content-length, it means that subsequent packet fragments have not arrived and waiting continues. If reassembly is not completed within a preset timeout period, it is considered that reassembly is complete. When reassembly is confirmed to be complete or timeout, an atomic renaming operation is performed to atomically rename the temporary file pointed to by fileID1 to fileID2. The atomic operation ensures that there is no intermediate state during file state migration, avoiding other processes from reading incomplete files. The renaming operation is completed through an atomic file operation function provided by the operating system. After renaming, the file path is migrated from the temporary path to the success path. After atomic renaming, the file path is migrated from the temporary path to the success path, updating the file storage location to ensure that the reassembled file can be correctly identified and processed by downstream modules. In the success path, the file is identified by fileID2, indicating that the file has completed reassembly and is ready for subsequent text restoration and analysis.

[0071] Step 6, the sink process judges whether the file exists and reads the text content assembly result data according to the fileID2 carried by the parsing result, if the file does not exist, the data is transmitted to the monitor process for asynchronous polling detection through the ring, the sink process receives the data from the process process in the DPDK ring queue, parses the data, extracts the key information including fileID2 and related metadata, and locates to the corresponding file path according to fileID2, and prepares for the next operation, according to fileID2 in the parsing result, checks whether the file exists, if the file exists, reads the file content, obtains the reassembled HTTP text data, assembles the text content and fileID2 into formatted data, and prepares to send to the middleware or upper module, if the file does not exist, checks whether the current time and the data recording time exceed the preset waiting time threshold, if the threshold is not exceeded, the data is sent to the monitor process through the DPDK ring, and asynchronous polling detection is requested, if the threshold is exceeded, the reassembly failure is marked, the failure information is assembled into formatted data, and sent to the middleware or upper module, according to whether the file exists and the reassembly state, the result data is processed, if the file exists and the reassembly is successful, the assembled formatted data is sent to the middleware or upper module, and the final output of the data is completed, if the file does not exist and needs asynchronous polling detection, the data is sent to the monitor process, and continues to wait for subsequent processing result, if the reassembly fails, the failure information is sent to the middleware or upper module, so as to carry out subsequent error processing or retry operation;

[0072] Step 7, the monitor process asynchronously polls the existence state of fileID2, notifies the sink process after completing reassembly, otherwise marks reassembly failure after timeout, the sink process reads file content according to fileID2, assembles into formatted data, and sends to middleware for use by upper layer modules, the monitor process receives data from the sink process in the DPDK ring queue, parses the data to obtain fileID2 and related metadata, locates the corresponding file path according to fileID2, and records the reception time of the data, at the same time, an asynchronous polling mechanism is started, and the existence of fileID2 is checked periodically, the polling interval can be adjusted according to system load and performance requirements, in the polling process, the existence state of fileID2 is checked, if the file exists, it is marked as reassembly success, the success state information and fileID2 are packaged, and sent back to the sink process through the DPDK ring, notifying that reassembly is completed, if the file does not exist, it is checked whether the current time exceeds the preset timeout threshold, if not, the polling is continued, if yes, it is marked as reassembly failure, the failure state information and fileID2 are packaged, and sent back to the sink process through the DPDK ring, notifying that reassembly fails, according to the polling result, the state information is sent back to the sink process, for the reassembly success state information, the sink process reads file content according to fileID2 after receiving the success state, assembles it with fileID2 into formatted data, and sends it to middleware for use by upper layer modules, for the reassembly failure state information, the sink process assembles the failure information into formatted data after receiving the failure state, and sends it to middleware for subsequent error handling or retry operation by upper layer modules.

[0073] As shown in Embodiment 2, on the basis of Embodiment 1, the application provides a technical solution: preferably, 1, in the process, complete flow collection and protocol analysis, support pcap / pcapng files and network card flow, socket communication and other ways to obtain data stream, the specific process is as follows: Figure 1 Figure 2 As shown in Embodiment 2, on the basis of Embodiment 1, the application provides a technical solution: preferably, 1, in the process, complete flow collection and protocol analysis, support pcap / pcapng files and network card flow, socket communication and other ways to obtain data stream, the specific process is as follows:

[0074] Process 1.1, when HTTP flow meeting the fragmentation feature is found, create or find hash node with five-tuple as key, generate file UUID according to timestamp and core id of process running, record file name fileID1:path-temp / UUID, copy a copy of message payload, record message tcp sequence number, and throw reassembly ring to reassembly process;

[0075] ​Flow 1.2, return to the original flow after non-blocking processing, record the file name fileID2:path-success / UUID, and throw the parsed data to the sink ring after processing is completed, which does not affect the performance of the module;

[0076] Flow 1.3, after receiving the same five-tuple packet again, if the judgment is successful, execute flow 1.1 to directly end, without continuing flow 1.2.

[0077] 2. In the reassembly process, data is collected from the process through the reassembly ring, and the offset is calculated according to the init TcpSeqNum of the five-tuple and the current TcpSeqNum of the packet, and the specific flow is as follows:

[0078] Flow 2.1, offset is 0, indicating that it is the first packet, a new file fileID1 is created and the packet payload is recorded.

[0079] Flow 2.2, non-first packet is recorded in the fileID1 file according to the offset, instead of directly appending the content, which solves the problems of out-of-order, packet loss and retransmission.

[0080] Flow 2.3, according to whether the length of fileID1 file is equal to the content-length of the packet, it is judged whether the restoration is completed, if it is completed or timeout, fileID1 (path-temp / UUID) is renamed to fileID2 (path-success / UUID), and the file path is moved.

[0081] 3. Sink process, data is received from process and monitor process through sink ring, and data is formatted and assembled according to the specification, if fileID2 field is found in the data, it is considered that the data is a fragmented flow, and it is checked whether the file corresponding to fileID2 exists, and the specific flow is as follows:

[0082] Flow 3.1, if the file exists, it means that the restoration has been completed, the file content is read, the restored text content and fileID2 are assembled together to form formatted data, which is sent to the middleware for use by the upper module.

[0083] Flow 3.2, if the file does not exist, it means that the restoration has not been completed, it is detected whether the actual time and the data recording time exceed the waiting time threshold, if they exceed, it means that the restoration fails, fileID2 is deleted, and the data is assembled into formatted data and sent to the middleware for use by the upper module.

[0084] Flow 3.3, if the file does not exist but the waiting time threshold has not been exceeded, the data is thrown to the monitor process through the monitor ring, without blocking the sink process.

[0085] 4. The monitor process receives the data through the monitor ring, checks whether the fileID2 file exists, and the specific flow is as follows:

[0086] Flow 4.1, if the file exists, the data is marked to indicate that the fileID2 restoration is successful, the sink process does not need to check again, and the data is thrown back to the sink ring. After the monitor receives the data, the formatted data is assembled according to the flow 3.1 scheme;

[0087] Flow 4.2, if the file does not exist and the waiting time threshold has been exceeded, the data is marked to indicate that the fileID2 restoration fails, the sink process does not need to check again, and the data is thrown back to the sink ring. After the monitor receives the data, the formatted data is assembled according to the flow 3.2 scheme;

[0088] Flow 4.3, if the file does not exist and the waiting time threshold has not been exceeded, the data is thrown back to the monitor ring, and the data is placed at the tail of the ring to repeat the flow 4.1 and 4.2.

[0089] The above is merely a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or replacements within the technical range disclosed by the present application, which shall be covered within the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. A DPDK-based HTTP text restoration method, characterized in that, The method comprises the following steps: Step 1, a DPDK-based multi-process architecture is constructed, and process, reassembly, sink and monitor processes are respectively run, and the processes are connected through a DPDK ring for asynchronous data transmission; Step 2, a TCP fragment HTTP stream is collected by the process, and a connection meeting a condition is parsed and recognized to generate a unique identifier fileID; Step 3, the fileID is named as fileID1 and is pointed to a temporary path, a payload and a TCP sequence number are extracted, and the payload and the TCP sequence number are sent to the reassembly process through the ring for recombination processing; Step 4, the reassembly process calculates an offset position of the payload in a target file according to the TCP sequence number, and writes the fileID1 file; Step 5, when a recombination completion condition is met, a file pointed by the fileID1 is atomically renamed as fileID2 and is moved to a success path; Step 6, the sink process judges whether a file exists according to the fileID2 carried by a parsing result, reads text content splicing result data, and if the file does not exist, transmits the data to the monitor process through the ring for asynchronous polling detection; Step 7, the monitor process asynchronously polls an existence state of the fileID2, notifies the sink process after recombination is completed, otherwise marks recombination failure after a timeout, the sink process reads file content according to the fileID2, splices the file content into formatted data, and sends the formatted data to a middleware for use by an upper module. 2.The HTTP text restoration method based on DPDK of claim 1, wherein: The step 1 specifically comprises: A system environment is initialized by using a DPDK framework, including configuration of a memory pool, creation of a CPU affinity setting, initialization of a ring structure of the DPDK for inter-process communication, and creation of four independent processes, namely, a process, a reassembly process, a sink process and a monitor process, wherein the process is responsible for traffic collection and protocol analysis, the reassembly process is responsible for fragmentation recombination and text restoration, the sink process is responsible for formatted message splicing, and the monitor process is responsible for text restoration timeout waiting; According to the designed architecture, the processes are deployed to different CPU cores, a DPDK ring structure is created for each process, the processes are connected through the DPDK ring for asynchronous data transmission, and a cooperative work flow is formed.

3. The HTTP text restoration method based on DPDK according to claim 1, characterized in that: The step 2 specifically comprises: The process acquires a TCP fragment HTTP stream in a network through multiple ways including a network card capture, a pcap file reading or a socket communication, and performs protocol analysis on the collected TCP fragments to recognize a message meeting a characteristic of an HTTP protocol; In the analysis process, the process process identifies each HTTP connection by analyzing the five-tuple of the TCP message, wherein each HTTP connection has its unique five-tuple identifier, and the five-tuple is the source IP address, the destination IP address, the source port, the destination port and the protocol type; For each identified HTTP connection, a unique identifier, i.e. fileID, is generated by the process process, which is composed of the five-tuple, the timestamp and the CPU core ID currently running by the process process, and then the generated fileID is recorded, and the TCP fragment message related to the HTTP connection is associated with the fileID and sent to the reassembly process for further processing through the DPDK ring structure.

4. The HTTP text restoration method based on DPDK according to claim 3, characterized in that: The fileID includes two states of fileID1 and fileID2, and the generation method of the fileID includes the UUID generated based on the TCP five-tuple, the timestamp and the core ID running by the process. The fileID1 is used to represent the intermediate file which has not completed reassembly, and is located in the temporary path. The fileID2 represents the file which has completed reassembly or only partially completed reassembly, and is located in the success path.

5. The DPDK-based HTTP text restoration method according to claim 4, characterized in that: The step 3 specifically includes: The original packet is captured by DPDK, the TCP header information is analyzed, the five-tuple is extracted, the HTTP connection state is maintained by using a hash table, if the five-tuple already exists, the last active time is updated, if it does not exist, a new connection context is created, the first identification timestamp and the current CPU core ID are recorded, and then the fileID1 is generated based on the five-tuple, the timestamp and the core ID; It is checked whether the TCP load contains the HTTP feature, the non-HTTP flow is filtered, and the key fields including the payload, the TCP sequence number and the fileID1 are extracted from the message, and then the payload and the sequence number are written into the temporary file, and the metadata is bound in the memory, if the message is a fragment, it is appended to the temporary file in sequence according to the sequence number; The DPDK Ring message structure is constructed, including the fileID1, the temporary file path and the sequence number range, and the message structure is sent to the reassembly process through the DPDK Ring.

6. The DPDK-based HTTP text restoration method according to claim 5, characterized in that: The step 4 specifically includes: The reassembly process receives the message structure from the process process in the DPDK ring queue, analyzes the message structure, extracts the fileID1 and the related data, locates to the corresponding temporary file path, and then checks whether the corresponding temporary file exists according to the fileID1, if the file does not exist, a new temporary file is created according to the fileID1, if the file already exists, the file is prepared for writing operation; According to the TCP sequence number, the offset position of the payload in the target file is calculated, wherein the TCP sequence number and the initial sequence number of the current packet are obtained from the message structure, the offset is calculated, and the payload is written into the corresponding position of the fileID1 file according to the calculated offset; if it is the first packet, offset = 0, the payload is written at the beginning of the file, and if it is a non-first packet, the payload is directly written into the specified position of the file according to the offset.

7. The DPDK-based HTTP text restoration method according to claim 6, characterized in that: The reassembly process calculates the offset offset according to the difference between the TCP sequence number and the initial sequence number, that is, the offset position is written, and the corresponding file of fileID1 is directly written; The expression of the offset is: ； In the formula, is an offset, indicating the offset position of the payload of the current packet in the target file, is a TCP sequence number, indicating the TCP sequence number of the current packet, is an initial sequence number, indicating the TCP sequence number of the first packet of the HTTP connection. 8.The HTTP text restoration method based on DPDK of claim 1, wherein: The step 5 specifically includes: In the reassembly process, after the writing of the payload of the current packet is completed, whether the recombination completion condition is met is checked by comparing whether the current length of the file is consistent with the expected content-length; if the file length is equal to the content-length, it is indicated that all data has been completely written, the recombination is completed, if the file length is less than the content-length, it is indicated that there is a subsequent fragment packet that has not arrived, the waiting is continued, and if the recombination is not completed within a preset timeout period, it is considered that the recombination is completed; When the recombination is completed or the timeout is confirmed, an atomic renaming operation is performed to atomically rename the temporary file pointed to by fileID1 to fileID2, wherein the renaming operation is completed by using an atomic file operation function provided by the operating system; after the renaming, the file path is migrated from the temporary path to the success path; After the atomic renaming is completed, the file path is migrated from the temporary path to the success path, the storage position of the file is updated, and in the success path, the file is identified by fileID2, indicating that the file has completed the recombination. 9.The HTTP text restoration method based on DPDK of claim 8, wherein: The step 6 specifically includes: The sink process receives the data from the process process from the DPDK ring queue, parses the data, extracts key information including fileID2 and related metadata, and locates to the corresponding file path according to fileID2, and prepares for the next operation; According to the fileID2 in the analysis result, whether the file exists is checked; if the file exists, the file content is read, the recombined HTTP text data is obtained, the text content is assembled with fileID2 into formatted data, and the formatted data is sent to the middleware or the upper module; if the file does not exist, whether the current time and the data recording time exceed a preset waiting time threshold is checked; if the threshold is not exceeded, the data is sent to the monitor process through the DPDK ring to request asynchronous polling detection; if the threshold is exceeded, the recombination failure is marked, the failure information is assembled into formatted data, and the formatted data is sent to the middleware or the upper module; According to the file existence and reorganization state, the processing result data is processed, if the file exists and the reorganization succeeds, the assembled formatted data is sent to the middleware or the upper module, the final output of the data is completed, if the file does not exist and needs to be asynchronously polled and detected, the data is sent to the monitor process, and subsequent processing results are continued to be waited, if the reorganization fails, the failure information is sent to the middleware or the upper module.

10. The HTTP text restoration method based on DPDK according to claim 9, characterized in that: The step 7 specifically comprises: The monitor process receives the data from the sink process in the DPDK ring queue, parses the data to obtain fileID2 and related metadata, positions to the corresponding file path according to fileID2, and records the receiving time of the data, simultaneously, an asynchronous polling mechanism is started, and whether the fileID2 file exists is periodically checked; In the polling process, the existence state of the fileID2 file is checked, if the file exists, the reorganization is marked as successful, the success state information and fileID2 are packaged, and are sent back to the sink process through the DPDK ring, the reorganization is notified to be completed, if the file does not exist, whether the current time and the data receiving time exceed a preset timeout threshold value is checked, if the timeout is not exceeded, the file is continued to be polled and waited, if the timeout is exceeded, the reorganization is marked as failed, the failure state information and fileID2 are packaged, and are sent back to the sink process through the DPDK ring, the reorganization is notified to fail; According to the polling result, the state information is sent back to the sink process, for the state information of the reorganization success, after the sink process receives the success state, the file content is read according to fileID2, is assembled with fileID2 as formatted data, and is sent to the middleware for the use of the upper module, for the state information of the reorganization failure, after the sink process receives the failure state, the failure information is assembled as formatted data, and is sent to the middleware for the subsequent error processing or retry operation of the upper module.

Images