A network data packet deduplication method and device based on DPDK

Abstract

The present application relates to a method and device for deduplicating network data packets based on DPDK, the method comprising obtaining and parsing the current data packet through a DPDK port; generating a corresponding to-be-verified Hash for the specified message content of the current data packet according to a preset format rule; wherein the to-be-verified Hash comprises a Hash basic table index, a Hash basic table data row index and a specified message content Hash; obtaining a reference data row from a memory according to the Hash basic table index and the Hash basic table data row index, and comparing the specified message content Hash with each slot Hash of the reference data row; if the comparison result satisfies the preset deduplication rule, discarding the current data packet indicated by the specified message content Hash. The present application can deduplicate duplicate data packets during network transmission while ensuring the integrity and security of the entire network communication, improving the transmission efficiency of network equipment, and reducing equipment performance pressure and bandwidth pressure during network transmission.

Description

Technical Field

[0001] The present application relates to the technical field of network data processing, and in particular to a method and device for deduplicating network data packets based on DPDK. Background Art

[0002] When network data packets are transmitted in the network data, some data packets that are only transmitted once appear twice or more during the network transmission process. When this situation occurs less frequently, it has no obvious impact on the transmission of network devices. However, when the number of duplicate data packets reaches a certain level, the network transmission will experience obvious congestion and delay. Therefore, it is particularly important to deduplicate network data packets, which can greatly improve the transmission efficiency of network devices and reduce the performance pressure and bandwidth pressure of devices during network transmission.

[0003] Therefore, how to propose a more effective method for deduplicating network data packets, which can deduplicate duplicate data packets during network transmission while ensuring the integrity and security of the entire network communication, is a technical problem that needs to be solved. Summary of the invention

[0004] The present application intends to provide a network data packet deduplication method, device, electronic device and storage medium based on DPDK. The technical problem to be solved is achieved through the following technical solution.

[0005] In a first aspect of this embodiment, a method for deduplicating network data packets based on DPDK is proposed, comprising:

[0006] Get the current data packet through the DPDK port and parse it;

[0007] According to the preset format rule, the specified message content of the current data packet generates a corresponding Hash to be verified; wherein the Hash to be verified includes a Hash basic table index, a Hash basic table data row index and a specified message content Hash;

[0008] Obtaining a reference data row from a memory according to the Hash basic table index and the Hash basic table data row index, and comparing the designated message content Hash with each slot Hash of the reference data row;

[0009] If the comparison result meets the preset deduplication rule, the current data packet indicated by the specified message content Hash is discarded.

[0010] Furthermore, obtaining the reference data row based on the Hash basic table index and the Hash basic table data row index also includes: checking each slot of the reference data row one by one, and if the slot identifier is valid and the difference between the slot record timestamp and the current timestamp exceeds a preset timeout threshold, updating the slot identifier to idle.

[0011] Furthermore, the preset deduplication rules include: when the specified message content Hash is equal to the current slot Hash, and the current slot is marked as valid, and the difference between the current slot record timestamp and the current timestamp does not exceed a preset timeout threshold, the current data packet indicated by the specified message content Hash is a duplicate data packet.

[0012] Furthermore, if the comparison result does not meet the preset deduplication rules, including:

[0013] When there is an idle slot in the reference data row, write the current data packet and the current timestamp into the idle slot, and update the slot identifier to be valid;

[0014] When there is no free slot in the reference data row, a Hash table overflow error is returned.

[0015] Furthermore, if the comparison result does not satisfy the preset deduplication rule, it also includes: writing the updated reference data row to the memory, and the memory forwards the current data packet.

[0016] Furthermore, the preset format rules are used to define the length and field format of the Hash to be verified; wherein, the length of the Hash to be verified is 64 bits, bits 0 to 19 are the data row index of the Hash basic table, bits 20 to 24 are the Hash basic table index, and bits 25 to 64 are the specified message content Hash; and / or, the specified message content Hash at least includes the Hash corresponding to the network message quintuple.

[0017] Furthermore, it also includes: when the current data packet is acquired, enabling the counter with a predefined period parameter; wherein the predefined period parameter is determined by the total number of rows in the basic table and the timing detection period, and the timing detection period is less than half of the maximum time length represented by the timestamp.

[0018] In a second aspect of this embodiment, a network data packet deduplication device based on DPDK is proposed, comprising:

[0019] The first module is configured to obtain the current data packet through the DPDK port and parse it;

[0020] The second module is configured to generate a corresponding Hash to be verified based on the specified message content of the current data packet according to a preset format rule; wherein the Hash to be verified includes a Hash basic table index, a Hash basic table data row index and a specified message content Hash;

[0021] A third module is configured to obtain a reference data row from a memory according to the Hash basic table index and / or the Hash basic table data row index, and compare the specified message content Hash with each slot Hash of the reference data row;

[0022] The fourth module is configured to discard the current data packet indicated by the specified message content Hash if the comparison result meets the preset deduplication rule.

[0023] In a third aspect of the present embodiment, an electronic device is proposed, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the processor implements the steps of the DPDK-based network data packet deduplication method as described in the first aspect.

[0024] In a fourth aspect of this embodiment, a computer-readable storage medium is proposed, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the DPDK-based network data packet deduplication method as described in the first aspect are implemented.

[0025] The embodiments of the present application include the following advantages:

[0026] The DPDK-based network data packet deduplication method provided in the embodiment of the present application obtains the current data packet through the DPDK port and parses it; generates a corresponding to-be-verified Hash for the specified message content of the current data packet according to the preset format rule; wherein the to-be-verified Hash includes a Hash basic table index, a Hash basic table data row index and a specified message content Hash; obtains a reference data row from the memory according to the Hash basic table index and the Hash basic table data row index, and compares the specified message content Hash with each slot Hash of the reference data row; if the comparison result meets the preset deduplication rule, discards the current data packet indicated by the specified message content Hash. The present application can deduplicate duplicate data packets during network transmission while ensuring the integrity and security of the entire network communication, improving the transmission efficiency of network equipment, and reducing equipment performance pressure and bandwidth pressure during network transmission. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 This is a schematic diagram of the principle of a network data packet deduplication method based on DPDK in an embodiment of the present application;

[0028] Figure 2 This is one of the flowcharts of a method for deduplicating network data packets based on DPDK in an embodiment of the present application;

[0029] Figure 3This is one of the data structure diagrams of a network data packet deduplication method based on DPDK in an embodiment of the present application;

[0030] Figure 4 This is the second data structure diagram of a network data packet deduplication method based on DPDK in an embodiment of the present application;

[0031] Figure 5 This is the third data structure diagram of a network data packet deduplication method based on DPDK in an embodiment of the present application;

[0032] Figure 6 It is a data field description table of a data structure of a network data packet deduplication method based on DPDK in an embodiment of the present application;

[0033] Figure 7 This is the second flowchart of a method for deduplicating network data packets based on DPDK in an embodiment of the present application;

[0034] Figure 8 This is the third flowchart of a method for deduplicating network data packets based on DPDK in an embodiment of the present application;

[0035] Fig. 9 This is a schematic diagram of the structure of a network data packet deduplication device based on DPDK according to an embodiment of the present application;

[0036] Fig.10 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0037] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0038] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0039] As described in the background technology, with the rapid development of network informatization, the amount of information flowing into the network link has increased dramatically, bringing great security risks to the intrusion detection, intrusion protection, and management and control of industrial, commercial, and national security. When a network data packet is transmitted in the network data, some remote packets that were originally transmitted only once appear twice or more during the network transmission process. When such a situation occurs in a small amount of data, there is no obvious impact on the transmission of network equipment, but when the number of such situations reaches a certain level, data transmission will show obvious congestion, delays, etc. Therefore, it is particularly important to deduplicate duplicate data packets during network transmission, which can greatly improve the transmission efficiency of network equipment and reduce the equipment performance pressure and bandwidth pressure during network transmission.

[0040] There are many ways to capture network packets, such as libpcap, pf-ring, DPDK, etc. Among them:

[0041] The libpcap method uses the traditional data packet acquisition method. After the network card receives the data packet, the first step is to copy the data from the network card to the main memory (RXbuffer). This copy process uses DMA (Direct Memory Access) technology. DMA transmission copies data from one address space to another address space. The CPU initializes this transmission action, and the transmission action itself is implemented and completed by the DMA controller. The CPU does not need to participate, and it will not consume CPU resources. Therefore, the copy in the first step can be ignored. In the second step, the kernel will copy the content of the network card that enters the main memory (RXbuffer) through DMA transmission for processing. Here, a data copy is performed. In the third step, since the memory of the kernel space and the user space is not shared, the kernel will copy the data packet content again for use in the user space. Therefore, libpcap will copy the data packet twice, that is, there are multiple copy operations and interrupt responses, which consumes CPU time slices and affects the overall data analysis capability of the system.

[0042] The pf-ring method uses mmap() to reduce the traditional two copies to one copy. The first step is to use DMA technology to copy the data from the network card to the main memory (RXbuffer). The second step is that pf-ring will copy the content of the network card that enters the main memory (RXbuffer) through DMA and put it into the ring buffer (ring). A data copy is performed here. The third step is to use mmap() to map the ring buffer to the user space, and the user space can directly access the data in this ring buffer. The packet capture and preprocessing of pf-ring are still completed in the kernel layer. Its disadvantage is that it increases the performance consumption of repeated scheduling during the data transmission with the user layer.

[0043] The DPDK method uses polling mode driver to replace the traditional mode of receiving and sending data packets through interrupted network cards. The received data packets are transferred to the memory through direct memory access mode DMA and directly handed over to the application for processing, thus achieving zero copy and greatly improving the performance of sending and receiving packets. The UIO mechanism is used to make the network card driver run in user mode, and the work originally processed in kernel mode is directly handed over to user mode application for processing, avoiding unnecessary system scheduling between kernel mode and user mode, and improving execution efficiency. DPDK binds 2MB or 1GB huge memory pages to replace traditional 4KB ordinary pages, improving memory usage efficiency, allowing programs to monopolize memory as much as possible to prevent memory swapping, expanding page tables to improve hash hit rate, and improving the page search rate in data capture. The main use of CPU affinity is to bind the control plane thread and each data plane thread to different CPU cores, saving the performance consumption of repeated scheduling, and supporting NUMA architecture to access local memory as much as possible. DPDK uses the rte_mbuf structure to store data packets, combining the data structure part and the data part together, so only one memory allocation is required, which further saves the memory allocation overhead and improves the speed of data reception and storage.

[0044] At present, most of the data capture and deduplication methods based on DPDK only use the DPDK packet capture interface to capture network data packets, simple hash algorithms and high-performance hardware to support the deduplication function, without considering the optimization of the overhead caused by the hash deduplication calculation itself. Therefore, this application mainly focuses on how to achieve algorithm acceleration after DPDK receives data packets and perform fast hash data packet deduplication under low cost, low memory, and low main frequency.

[0045] Embodiment 1:

[0046] like Figure 1 As shown, it is a schematic diagram of the principle of a network data packet deduplication method based on DPDK according to an embodiment of the present application. Since multiple RX Queues can be bound to different CPU cores for processing through hard interrupts when using the DPDK method, it is possible to achieve balanced utilization of CPU resources and improve network throughput.

[0047] like Figure 2 FIG. 1 is a flow chart of a method for deduplicating network data packets based on DPDK according to an embodiment of the present application. The method for deduplicating network data packets based on DPDK comprises:

[0048] S101: Obtain the current data packet through the DPDK port and parse it.

[0049] S102: Generate a corresponding Hash to be verified from the specified message content of the current data packet according to a preset format rule; wherein the Hash to be verified includes a Hash basic table index, a Hash basic table data row index and a specified message content Hash.

[0050] S103: Obtain a reference data row from a memory according to the Hash basic table index and the Hash basic table data row index, and compare the designated message content Hash with each slot Hash of the reference data row.

[0051] S104: If the comparison result satisfies the preset deduplication rule, the current data packet indicated by the specified message content Hash is discarded.

[0052] In some embodiments, the preset format rule is used to define the length and field format of the Hash to be verified; wherein the length of the Hash to be verified is 64 bits, bits 0 to 19 are the Hash basic table data row index, bits 20 to 24 are the Hash basic table index, and bits 25 to 64 are the specified message content Hash. For example, Hash basic table data row index Hslotindex = H[19:0]; Hash basic table index Htid = [24:20]; specified message content HashHhid = H[63:25].

[0053] In some embodiments, the specified message content Hash includes at least the Hash corresponding to the five-tuple of the network message. It should be noted that, in order to improve performance, the Hash calculation part can consider only performing Hash calculation on the five-tuple part; for the message content part, only part of the data is sampled, or Hash calculation is performed after sampling, avoiding Hash calculation on the entire packet.

[0054] In some embodiments, obtaining the reference data row based on the Hash basic table index and the Hash basic table data row index also includes: checking each slot of the reference data row one by one, and if the slot identifier is valid and the difference between the slot record timestamp and the current timestamp exceeds a preset timeout threshold, updating the slot identifier to free.

[0055] Specifically, first, according to the Hash basic table index Htid corresponding to the Hash basic table, then from the table according to the Hash basic table data row index Hslotindex to obtain the corresponding reference data row, recorded as Dslot. Then, check each slot in the current row Dslot one by one. If the slot is marked as valid and the difference between the slot record timestamp and the current timestamp exceeds the preset timeout threshold, the slot mark is updated to free. Reference Figures 3 to 6The Hash basic table data structure diagram and data structure description table of each data domain, SlotAx represents the slot (A0-A10) in the multi-slot Hash table. Each row of the Hash table has 10 slots for data insertion to prevent data loss caused by Hash conflicts. Slot_Ecc is used as the ecc check for each row of the Hash table for error detection, which plays an accelerating role to a certain extent. Prevent erroneous data from traversing each slot. And when the cumulative Ecc count error reaches a certain threshold, the row is initialized to prevent the accumulation of write errors and consume resources. The secondary domain is the data element of each primary domain data block, which stores specific elements in the data. Structuring facilitates Hash data comparison. Figure 6 The b unit is bit, and the B unit is Byte.

[0056] In some embodiments, the preset deduplication rules include: when the specified message content Hash is equal to the current slot Hash, and the current slot is marked as valid, and the difference between the current slot record timestamp and the current timestamp does not exceed a preset timeout threshold, the current data packet indicated by the specified message content Hash is a duplicate data packet.

[0057] In some embodiments, if the comparison result does not satisfy the preset deduplication rule, such as Figure 7 As shown, including:

[0058] S711: When there is an idle slot in the reference data row, write the current data packet and the current timestamp into the idle slot, and update the slot identifier to be valid.

[0059] S712: When there is no free slot in the reference data row, a Hash table overflow error is returned.

[0060] In some embodiments, if the comparison result does not satisfy the preset deduplication rule, it also includes: writing the updated reference data row to the memory, and the memory forwards the current data packet.

[0061] In some embodiments, in order to ensure that the network data packet deduplication operation does not affect the network data transmission efficiency, it is necessary to set up a timeout detection and timeout item removal mechanism. The timeout detection should be performed immediately when the non-duplicate data packet is inserted, and at the same time, it should prevent the counter "overflow / wrap-around" phenomenon of the table items that have not been accessed for a long time from causing errors. Therefore, in order to prevent the counter "overflow / wrap-around" phenomenon of the table items that have not been accessed for a long time from causing errors, such as Figure 8 As shown, the method also includes:

[0062] S105: When the current data packet is acquired, a counter is enabled with a predefined period parameter; wherein the predefined period parameter is determined by the total number of rows in the basic table and a timing detection period, and the timing detection period is less than half of the maximum representation time length of the timestamp.

[0063] Specifically, the timeout detection and timeout item removal process can be described as follows: when the Hash basic table index Htid = 0 and the Hash basic table data row index Hslotindex = 0 is initialized, the timer is set and started, and the timing period is set to s; when a single task is started, the corresponding reference data row Dslot is obtained according to the Hash basic table index Htid and the Hash basic table data row index Hslotindex, and each slot in the current reference data row Dslot is checked one by one. If the slot is valid and the difference between the timestamp and the current timestamp exceeds the timeout threshold, the valid flag of the slot is cleared and set to an idle slot; otherwise, the current row Dslot is written back. When the Hash basic table index Htid or the Hash basic table data row index Hslotindex reaches the maximum value, it is set to 0 respectively.

[0064] It should be noted that the above timing parameter s should be determined by the total number of rows in the Hash basic table and the timing detection period. In order to prevent the "loopback / overflow" problem caused by the timestamp timer bit width problem, the timing detection period should be less than half of the maximum time length of the timestamp. For example, if the timestamp timer bit width is 16 bits (the timing unit is milliseconds), the maximum time length is about 65 seconds, and it is recommended that the timing detection period does not exceed 30 seconds.

[0065] In addition, a pipeline design structure can be used in the specific implementation of this embodiment. First, using a single pipeline is more convenient for preserving the order of data packets than multi-channel parallelism. Second, the core of improving performance lies in hiding DDR access delays, which is easier to achieve in a pipeline. Third, the computing load of each CPU core is as roughly equal as possible, which makes it easier to improve the overall throughput. Fourth, the buffer data depth between each level of the pipeline needs to be controlled by parameters, which makes it easier to tune the performance through parameter adjustment. Fifth, in order to improve performance, for the hash calculation part, you can consider performing hash calculations on the quintuple part, but only sampling part of the data for the content part, or performing hash calculations after sampling. Avoid hash calculations on the entire packet.

[0066] The DPDK-based network data packet deduplication method provided in the embodiment of the present application obtains the current data packet through the DPDK port and parses it; generates a corresponding to-be-verified Hash for the specified message content of the current data packet according to the preset format rule; wherein the to-be-verified Hash includes a Hash basic table index, a Hash basic table data row index and a specified message content Hash; obtains a reference data row from the memory according to the Hash basic table index and the Hash basic table data row index, and compares the specified message content Hash with each slot Hash of the reference data row; if the comparison result meets the preset deduplication rule, discards the current data packet indicated by the specified message content Hash. The present application can deduplicate duplicate data packets during network transmission while ensuring the integrity and security of the entire network communication, improving the transmission efficiency of network equipment, and reducing equipment performance pressure and bandwidth pressure during network transmission.

[0067] Embodiment 2:

[0068] Fig. 9 Schematic diagram of a network data packet deduplication device based on DPDK provided in an embodiment of the present application. Fig. 9 As shown, the device comprises:

[0069] The first module 901 is configured to obtain and parse the current data packet through the DPDK port.

[0070] The second module 902 is configured to generate a corresponding Hash to be verified based on the specified message content of the current data packet according to a preset format rule; wherein the Hash to be verified includes a Hash basic table index, a Hash basic table data row index and a specified message content Hash.

[0071] The third module 903 is configured to obtain a reference data row from the memory according to the Hash basic table index and / or the Hash basic table data row index, and compare the designated message content Hash with each slot Hash of the reference data row.

[0072] The fourth module 904 is configured to discard the current data packet indicated by the specified message content Hash if the comparison result meets the preset deduplication rule.

[0073] The fifth module 905 is configured to enable the counter with a predefined period parameter when the current data packet is acquired; wherein the predefined period parameter is determined by the total number of rows in the basic table and the timing detection period, and the timing detection period is less than half of the maximum time length represented by the timestamp.

[0074] It should be understood that the DPDK-based network data packet deduplication device of the embodiment of this specification can also execute Figures 1 to 8 A method for executing a network data packet deduplication device based on DPDK, and implementing a network data packet deduplication device based on DPDK in Figures 1 to 8 The functions of the example shown will not be described in detail here.

[0075] At the same time, the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0076] Embodiment three:

[0077] Fig.10 Schematic diagram of an electronic device 10 provided in an embodiment of the present application. Fig.10 As shown, the electronic device 10 of this embodiment includes: a processor 1001, a memory 1002, and a computer program 1003 stored in the memory 1002 and executable on the processor 1001. When the processor 1001 executes the computer program 1003, the steps in the above-mentioned method embodiments are implemented. Alternatively, when the processor 1001 executes the computer program 1003, the functions of the modules / units in the above-mentioned device embodiments are implemented.

[0078] The electronic device 10 may be a desktop computer, a notebook, a PDA, a cloud server, or other electronic device. The electronic device 10 may include, but is not limited to, a processor 1001 and a memory 1002. Those skilled in the art will appreciate that Fig.10 The electronic device 10 is merely an example and does not limit the electronic device 10 . The electronic device 10 may include more or less components than those shown in the figure, or different components.

[0079] The memory 1002 may be an internal storage unit of the electronic device 10, for example, a hard disk or memory of the electronic device 10. The memory 1002 may also be an external storage device of the electronic device 10, for example, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card (FlashCard), etc. equipped on the electronic device 10. The memory 1002 may also include both an internal storage unit of the electronic device 10 and an external storage device. The memory 1002 is used to store computer programs and other programs and data required by the electronic device.

[0080] The processor 1001 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor 1001 reads the corresponding computer program from the non-volatile memory into the memory and then runs it, forming a shared resource access control device at the logical level. The processor executes the program stored in the memory, and is specifically used to perform the following operations:

[0081] Get the current data packet through the DPDK port and parse it;

[0082] According to the preset format rule, the specified message content of the current data packet generates a corresponding Hash to be verified; wherein the Hash to be verified includes a Hash basic table index, a Hash basic table data row index and a specified message content Hash;

[0083] Obtaining a reference data row from a memory according to the Hash basic table index and the Hash basic table data row index, and comparing the designated message content Hash with each slot Hash of the reference data row;

[0084] If the comparison result satisfies the preset deduplication rule, the current data packet indicated by the specified message content Hash is discarded;

[0085] When the current data packet is acquired, the counter is enabled with a predefined period parameter; wherein the predefined period parameter is determined by the total number of rows in the basic table and a timing detection period, and the timing detection period is less than half of the maximum representation time length of the timestamp.

[0086] The above is as in this manual Figures 1 to 8The DPDK-based network data packet deduplication method disclosed in the illustrated embodiment can be applied to the processor 1001, or implemented by the processor 1001. The processor 1001 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in the processor or an instruction in the form of software. The above processor can implement or execute the various methods, steps and logic block diagrams disclosed in the embodiments of this specification. The general processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiments of this specification can be directly embodied as a hardware decoding processor to perform, or a combination of hardware and software modules in the decoding processor to perform. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0087] Of course, in addition to software implementation, the electronic device of the embodiments of this specification does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0088] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units.

[0089] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. The computer program may include computer program code, and the computer program code may be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium does not include electric carrier signals and telecommunication signals.

[0090] Embodiment 4:

[0091] The embodiment of the present specification also provides a computer-readable storage medium, which stores one or more programs, wherein the one or more programs include instructions, which, when executed by a portable electronic device including a plurality of application programs, enable the portable electronic device to execute Figures 1 to 8 The DPDK-based network data packet deduplication method of the illustrated embodiment is specifically used to perform the following method:

[0092] Get the current data packet through the DPDK port and parse it;

[0093] According to the preset format rule, the specified message content of the current data packet generates a corresponding Hash to be verified; wherein the Hash to be verified includes a Hash basic table index, a Hash basic table data row index and a specified message content Hash;

[0094] Obtaining a reference data row from a memory according to the Hash basic table index and the Hash basic table data row index, and comparing the designated message content Hash with each slot Hash of the reference data row;

[0095] If the comparison result satisfies the preset deduplication rule, the current data packet indicated by the specified message content Hash is discarded;

[0096] When the current data packet is acquired, the counter is enabled with a predefined period parameter; wherein the predefined period parameter is determined by the total number of rows in the basic table and a timing detection period, and the timing detection period is less than half of the maximum representation time length of the timestamp.

[0097] In short, the above is only a preferred embodiment of this specification, and is not intended to limit the protection scope of this specification. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of this specification should be included in the protection scope of this specification.

[0098] It should be noted that the above detailed descriptions are exemplary and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those of ordinary skill in the art to which the present application belongs.

[0099] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should also be understood that when the terms "comprise" and / or "include" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0100] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein.

[0101] In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units that are not explicitly listed or inherent to these processes, methods, products, or apparatuses.

[0102] For ease of description, spatially relative terms, such as "above", "above", "on the upper surface of", "above", etc., may be used herein to describe the spatial positional relationship between a device or feature and other devices or features as shown in the figure. It should be understood that spatially relative terms are intended to include different orientations of the device in use or operation in addition to the orientation described in the figure. For example, if the device in the accompanying drawings is inverted, the device described as "above other devices or structures" or "above other devices or structures" will be positioned as "below other devices or structures" or "below other devices or structures". Thus, the exemplary term "above" may include both "above" and "below". The device may also be positioned in other different ways, such as rotated 100 degrees or in other orientations, and the spatially relative descriptions used herein are interpreted accordingly.

[0103] In the above detailed description, reference is made to the accompanying drawings, which form a part of this document. In the accompanying drawings, similar symbols typically identify similar components unless the context indicates otherwise. The illustrated embodiments described in the detailed description, drawings, and claims are not meant to be limiting. Other embodiments may be used, and other changes may be made, without departing from the spirit or scope of the subject matter presented herein.

[0104] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A network data packet deduplication method based on DPDK, characterized in that: include: Get the current data packet through the DPDK port and parse it; According to the preset format rule, the specified message content of the current data packet generates a corresponding Hash to be verified; wherein the Hash to be verified includes a Hash basic table index, a Hash basic table data row index and a specified message content Hash; According to the Hash basic table index and the Hash basic table data row index, a reference data row is obtained from the memory, including: checking each slot of the reference data row one by one, if the slot identifier is valid and the difference between the slot record timestamp and the current timestamp exceeds a preset timeout threshold, then updating the slot identifier to idle, and comparing the specified message content Hash with each slot Hash of the reference data row; If the comparison result satisfies the preset deduplication rule, the current data packet indicated by the specified message content Hash is discarded; the preset deduplication rule includes: when the specified message content Hash is equal to the current slot Hash, and the current slot is marked as valid, and the difference between the current slot record timestamp and the current timestamp does not exceed the preset timeout threshold, the current data packet indicated by the specified message content Hash is a duplicate data packet; If the comparison result does not meet the preset deduplication rules, including: When there is an idle slot in the reference data row, write the current data packet and the current timestamp into the idle slot, and update the slot identifier to be valid; When there is no free slot in the reference data row, a Hash table overflow error is returned; When the current data packet is obtained, the counter is enabled with a predefined cycle parameter; wherein the predefined cycle parameter is determined by the total number of rows in the basic table and the timing detection cycle, and the timing detection cycle is less than half of the maximum time length represented by the timestamp; during initialization, that is, when the Hash basic table index Htid=0 and the Hash basic table data row index Hslotindex=0, the timer is set and started, and the timing cycle is set to s at this time; when a single task is started, the corresponding reference data row Dslot is obtained according to the Hash basic table index Htid and the Hash basic table data row index Hslotindex, and each slot in the current reference data row Dslot is checked one by one. If the slot is valid and the difference between the timestamp and the current timestamp exceeds the timeout threshold, the valid flag of the slot is cleared and set to an idle slot; otherwise, the current row Dslot is written back; when the Hash basic table index Htid or the Hash basic table data row index Hslotindex reaches the maximum value, it is set to 0 respectively; the timing cycle s is determined by the two parameters of the total number of rows in the Hash basic table and the timing detection cycle.

2. The DPDK-based network data packet deduplication method according to claim 1, characterized in that: If the comparison result does not satisfy the preset deduplication rule, the method further includes: writing the updated reference data row to the memory, and the memory forwards the current data packet.

3. The DPDK-based network data packet deduplication method according to any one of claims 1 to 2, characterized in that: The preset format rule is used to define the length and field format of the Hash to be verified; wherein, the length of the Hash to be verified is 64 bits, bits 0 to 19 are the data row index of the Hash basic table, bits 20 to 24 are the Hash basic table index, and bits 25 to 64 are the specified message content Hash; and / or, the specified message content Hash at least includes the Hash corresponding to the network message quintuple.

4. A DPDK-based network data packet deduplication device for implementing the DPDK-based network data packet deduplication method as described in any one of claims 1 to 3, characterized in that: include: The first module is configured to obtain the current data packet through the DPDK port and parse it; The second module is configured to generate a corresponding Hash to be verified based on the specified message content of the current data packet according to a preset format rule; wherein the Hash to be verified includes a Hash basic table index, a Hash basic table data row index and a specified message content Hash; A third module is configured to obtain a reference data row from a memory according to the Hash basic table index and / or the Hash basic table data row index, and compare the specified message content Hash with each slot Hash of the reference data row; The fourth module is configured to discard the current data packet indicated by the specified message content Hash if the comparison result meets the preset deduplication rule.

5. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When executing the computer program, the processor implements the steps of the DPDK-based network data packet deduplication method according to any one of claims 1 to 3.

6. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the DPDK-based network data packet deduplication method according to any one of claims 1 to 3 are implemented.

Images