A flow deduplication method, device, electronic device and storage medium

Abstract

The present invention provides a traffic deduplication method, device, electronic device and storage medium, and relates to the field of information technology. The method comprises: obtaining target data sent by a sender, slicing the target data to obtain a slice data packet list, obtaining the transmission bandwidth of the target data, if the transmission bandwidth of the target data does not meet the preset total bandwidth ratio requirement, marking the slice data packet based on a preset deduplication algorithm, obtaining an initial deduplication data packet list and an initial learning data packet list, performing network traffic deduplication monitoring for the target data in a preset time period, obtaining a network traffic deduplication monitoring result, if the network traffic deduplication monitoring result meets the traffic deduplication monitoring requirement, using a first detection algorithm to obtain a final deduplication triplet list, thereby achieving deduplication of the target data, obtaining the deduplicated target data for transmission, and improving the traffic deduplication transmission efficiency.

Description

Technical Field

[0001] The present invention relates to the field of information technology, and in particular to a traffic deduplication method, device, electronic equipment and storage medium. Background Art

[0002] Network traffic deduplication refers to the technology of checking whether some data has been transmitted when transmitting data. For some data that has been transmitted, only a mark is given and the corresponding data packet is no longer transmitted to reduce the transmission volume.

[0003] When data is communicated, a triplet is used to uniquely identify a process in the data communication. This triplet is called a semi-correlation. The semi-correlation specifies each half of the communication connection. The triplet includes: IP address, port number, and communication protocol.

[0004] Network traffic deduplication is also called data deduplication or data deduplication. Network deduplication can reduce the amount of data sent over the network, thereby reducing bandwidth usage costs and improving network efficiency. Since the amount of data transmitted is reduced, the data transmission speed is accelerated. Therefore, network deduplication is widely used in scenarios such as data backup, archiving, synchronization or data analysis. Therefore, how to efficiently deduplicate traffic has been continuously developing. Summary of the invention

[0005] In view of the above technical problems, the technical solution adopted by the present invention is: a traffic deduplication method, the method is used for transmitting target data from a sender to a receiver, and the method includes the following contents:

[0006] Obtain target data sent by a sender, and slice the target data to obtain a slice data packet list, wherein the slice data packet list includes a plurality of slice data packets;

[0007] Obtaining the transmission bandwidth of the target data, if the transmission bandwidth of the target data does not meet the preset total bandwidth ratio requirement, identifying the slice data packets based on the preset deduplication algorithm, and obtaining an initial deduplication data packet list and an initial learning data packet list, wherein the initial deduplication data packet list includes a number of initial deduplication data packets with a first identifier, and the initial learning data packet list includes a number of initial learning data packets with a second identifier, wherein the first identifier is different from the second identifier;

[0008] Perform network traffic deduplication monitoring on target data in a preset time period to obtain network traffic deduplication monitoring results;

[0009] If the network traffic deduplication monitoring result meets the traffic deduplication monitoring rule, use the first detection algorithm to obtain the final deduplication triplet list, and use the mapping value of the slice data packet corresponding to the final deduplication triplet to replace the slice data packet corresponding to the final deduplication triplet, so as to obtain the deduplicated target data for transmission, and the final deduplication triplet list includes several final deduplication triplets;

[0010] The first detection algorithm comprises the following steps:

[0011] Obtaining a first election triplet list, wherein the first election triplet list includes a plurality of first election triplets, and the first election triplet is a communication identifier corresponding to an initial deduplication data packet;

[0012] If the deduplication rate corresponding to the first election triplet list meets the preset deduplication rule, the first election triplet list is determined as the final deduplication triplet list; otherwise, an alternative triplet list is obtained, wherein the alternative triplet list includes a plurality of alternative triplets, and the alternative triplets are communication identifiers corresponding to the initial learning data packets that meet the pre-equipped selection rule, and the preset deduplication rule is stricter than the traffic deduplication monitoring rule;

[0013] If there is a candidate triplet with a bandwidth greater than a first election triplet, the first election triplet is replaced by the candidate triplet to obtain a list of first election triplet after replacement;

[0014] If the deduplication rate corresponding to the replaced first election triple list meets the preset deduplication rule, the replaced first election triple list is determined as the final deduplication triple list; otherwise, the previous step is executed.

[0015] A flow deduplication device, comprising:

[0016] A target data acquisition module is used to acquire target data sent by a sender, and slice the target data to acquire a slice data packet list, wherein the slice data packet list includes a plurality of slice data packets;

[0017] A preset deduplication algorithm module is used to obtain the transmission bandwidth of the target data. If the transmission bandwidth of the target data does not meet the preset total bandwidth ratio requirement, the slice data packet is marked based on the preset deduplication algorithm to obtain an initial deduplication data packet list and an initial learning data packet list, wherein the initial deduplication data packet list includes a plurality of initial deduplication data packets with a first mark, and the initial learning data packet list includes a plurality of initial learning data packets with a second mark, wherein the first mark is different from the second mark;

[0018] A network traffic deduplication monitoring module is used to perform network traffic deduplication monitoring on target data for a first preset time period and obtain network traffic deduplication monitoring results;

[0019] A first detection algorithm module is used to obtain a final deduplication triplet list using the first detection algorithm if the network traffic deduplication monitoring result meets the traffic deduplication monitoring rule, and replace the slice data packet corresponding to the final deduplication triplet with the mapping value of the slice data packet corresponding to the final deduplication triplet to obtain the deduplicated target data for transmission, wherein the final deduplication triplet list includes several final deduplication triplets;

[0020] Wherein, the first detection algorithm module includes the following submodules:

[0021] A first election triplet acquisition submodule, used to acquire a first election triplet list, wherein the first election triplet list includes a plurality of first election triplets, and the first election triplet is a communication identifier corresponding to an initial deduplication data packet;

[0022] The first connection submodule is used to determine the first election triplet list as the final deduplication triplet list if the deduplication rate corresponding to the first election triplet list meets the preset deduplication rule; otherwise, obtain an alternative triplet list, wherein the alternative triplet list includes a plurality of alternative triplets, and the alternative triplets are communication identifiers corresponding to the initial learning data packets that meet the pre-equipped selection rule, and the preset deduplication rule is stricter than the traffic deduplication monitoring rule;

[0023] A replacement submodule is used for replacing the first election triplet with the candidate triplet if the bandwidth of the candidate triplet is greater than the bandwidth of the first election triplet, and obtaining a list of first election triplet after replacement;

[0024] The second connection submodule is used to determine the replaced first election triple list as the final deduplicated triple list if the deduplication rate corresponding to the replaced first election triple list meets the preset deduplication rule; otherwise, execute the replacement submodule.

[0025] According to another aspect of the present invention, a non-transitory computer-readable storage medium is provided, in which at least one instruction or at least one program is stored, and the at least one instruction or the at least one program is loaded and executed by a processor to implement the aforementioned method.

[0026] According to yet another aspect of the present invention, an electronic device is provided, comprising a processor and the aforementioned non-transitory computer-readable storage medium.

[0027] The present invention has at least the following beneficial effects: In summary, the target data sent by the sender is obtained, and the target data is sliced ​​to obtain a slice data packet list, the transmission bandwidth of the target data is obtained, if the transmission bandwidth of the target data does not meet the preset total bandwidth ratio requirement, the slice data packets are identified based on the preset deduplication algorithm, an initial deduplication data packet list and an initial learning data packet list are obtained, network traffic deduplication monitoring is performed on the target data for a preset time period, and the network traffic deduplication monitoring result is obtained. If the network traffic deduplication monitoring result meets the traffic deduplication monitoring requirement, a first detection algorithm is used to obtain a final deduplication triplet list, thereby achieving deduplication of the target data, and the deduplicated target data is obtained for transmission. The present invention deduplicates the slice data packets and transmits them after deduplication, thereby improving the traffic deduplication transmission efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0029] Figure 1 A flow chart of a traffic deduplication method provided by an embodiment of the present invention;

[0030] Figure 2 A flowchart of a first detection algorithm provided by an embodiment of the present invention;

[0031] Figure 3 A flowchart of S400 provided in an embodiment of the present invention;

[0032] Figure 4 A flowchart of the steps performed by a receiver after receiving the final deduplicated data provided by an embodiment of the present invention;

[0033] Figure 5 A schematic diagram of the structure of a flow deduplication device provided by an embodiment of the present invention;

[0034] Figure 6 A schematic diagram of the structure of a first detection algorithm module provided in an embodiment of the present invention;

[0035] Figure 7 A schematic diagram of the structure of a sender deduplication submodule provided in another embodiment of the present invention;

[0036] Figure 8 A schematic structural diagram of a receiver deduplication submodule provided in another embodiment of the present invention. DETAILED DESCRIPTION

[0037] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0038] Embodiment 1

[0039] Embodiment 1 of the present invention provides a traffic deduplication method, wherein the method is used to transmit target data from a sender to a receiver, such as Figure 1 As shown, the method comprises the following steps:

[0040] S100, obtaining target data sent by a sender, and slicing the target data to obtain a slice data packet list, wherein the slice data packet list includes a plurality of slice data packets.

[0041] Specifically, those skilled in the art know that any method of slicing data in the prior art falls within the protection scope of the present invention and will not be described in detail here.

[0042] S200, obtaining the transmission bandwidth of the target data. If the transmission bandwidth of the target data does not meet the preset total bandwidth ratio requirement, the slice data packets are identified based on the preset deduplication algorithm to obtain an initial deduplication data packet list and an initial learning data packet list, wherein the initial deduplication data packet list includes several initial deduplication data packets with a first identification, and the initial learning data packet list includes several initial learning data packets with a second identification, wherein the first identification is different from the second identification.

[0043] In a specific embodiment of the present invention, the transmission bandwidth of the target data meets the preset total bandwidth ratio requirement: the transmission bandwidth of the target data divided by the total transmission bandwidth is greater than the preset bandwidth ratio threshold a, wherein the preset bandwidth ratio threshold can be determined according to actual needs. Optionally, 0.2<a<0.4; preferably, a=0.3. Detection is started only when the transmission bandwidth of the target data meets certain requirements.

[0044] Specifically, the preset deduplication algorithm is the MAXP algorithm, and the sliced ​​data packets are identified using the MAXP algorithm provided by the paper (Ashok Anand, ChitraMuthukrishnan, Aditya Akella, Ramachandran Ramjee, Redundancy in NetworkTraffic: Findings and Implications, University of Wisconsin-Madison, †MicrosoftResearch India).

[0045] Specifically, the MAXP algorithm is used to identify the sliced ​​data packets and obtain the initial deduplication data packet list A={A 1 , A 2 , …, A i , …, A m} and the initial learning data packet list B = {B 1 , B 2 , …, B j , …, B n}, the i-th initial deduplication data packet A i is the sliced ​​data packet marked as "de-duplication flag" by the MAXP algorithm. The value range of i is 1 to m, where m is the number of initial de-duplication data packets. The jth initial learning data packet B j It is a slice data packet marked as "learning mark" by the MAXP algorithm. The value range of j is 1 to n, and n is the number of initial learning data packets. Among them, m+n is not greater than the number of slice data packets.

[0046] S300, performing network traffic deduplication monitoring on target data in a preset time period to obtain a network traffic deduplication monitoring result.

[0047] Specifically, network traffic deduplication monitoring is performed on the target data for a preset time period t. Any network traffic deduplication monitoring in the prior art falls within the protection scope of the present invention and will not be described in detail here.

[0048] Furthermore, the network traffic deduplication monitoring result is a deduplication rate within a preset time period t.

[0049] In one embodiment of the present invention, t is the time that the deduplication buffer can store data for executing b times.

[0050] In another embodiment of the present invention, t is determined based on the size of the stored data in the deduplication buffer. Specifically, t is the time to execute b times the stored data in the deduplication buffer. It can be understood that the size of the stored data in the deduplication buffer changes based on the transmission bandwidth of the target data, which is equivalent to the preset time period t changing based on the change of the transmission bandwidth, so that the value of t is more reasonable and more efficient.

[0051] Specifically, the purpose of conducting network traffic deduplication monitoring during a preset time period is to determine the value of network traffic deduplication before using the first detection algorithm for detection; if the network traffic deduplication monitoring results meet the traffic deduplication monitoring requirements, it is considered valuable to perform deduplication using the first detection algorithm at this time, and network traffic deduplication monitoring during a preset time period is adopted to avoid redundant work.

[0052] S400, if the network traffic deduplication monitoring results meet the traffic deduplication monitoring rules, use the first detection algorithm to obtain the final deduplication triplet list, and use the mapping value of the slice data packet corresponding to the final deduplication triplet to replace the slice data packet corresponding to the final deduplication triplet, so as to obtain the deduplicated target data for transmission, and the final deduplication triplet list contains several final deduplication triplets.

[0053] Specifically, if the network traffic deduplication monitoring result meets the traffic deduplication monitoring requirements, it can be understood that the deduplication rate in the preset time period t is greater than the first preset deduplication rate.

[0054] Specifically, the detection size of the first detection algorithm is a preset multiple b of the deduplication buffer size. Optionally, 3≤b≤5; preferably, b=4.

[0055] In summary, the target data sent by the sender is obtained, and the target data is sliced ​​to obtain a slice data packet list, and the transmission bandwidth of the target data is obtained. If the transmission bandwidth of the target data does not meet the preset total bandwidth ratio requirement, the slice data packets are identified based on the preset deduplication algorithm, and an initial deduplication data packet list and an initial learning data packet list are obtained. The network traffic deduplication monitoring for the target data in a preset time period is performed to obtain the network traffic deduplication monitoring result. If the network traffic deduplication monitoring result meets the traffic deduplication monitoring requirement, the first detection algorithm is used to obtain the final deduplication triplet list, thereby achieving deduplication of the target data, and the deduplicated target data is obtained for transmission. The present invention deduplicates the slice data packets and transmits them after deduplication, thereby improving the traffic deduplication transmission efficiency.

[0056] Among them, Figure 2 As shown, the first detection algorithm includes the following steps:

[0057] S410, obtaining a first election triplet list, wherein the first election triplet list includes a plurality of first election triplets, and the first election triplet is a communication identifier corresponding to an initial deduplication data packet.

[0058] Specifically, the first election triplet is (destination IP address, transmission port, transmission protocol) or (source IP address, transmission port, transmission protocol) of the corresponding initial deduplication data packet.

[0059] In one embodiment of the present invention, it also includes: defining the type of traffic deduplication, when the type of traffic deduplication is the first deduplication method, the first selection triplet is the (target IP address, port, transmission protocol) of the corresponding initial deduplication data packet; when the type of traffic deduplication is the second deduplication method, the first selection triplet is the (source IP, port, transmission protocol) of the corresponding initial deduplication data packet.

[0060] S420, if the deduplication rate corresponding to the first election triplet list meets the preset deduplication rules, the first election triplet list is determined as the final deduplication triplet list; otherwise, an alternative triplet list is obtained, wherein the alternative triplet list contains several alternative triplets, and the alternative triplets are communication identifiers corresponding to the initial learning data packets that meet the pre-equipped selection rules, and the preset deduplication rules are stricter than the traffic deduplication monitoring rules.

[0061] Specifically, the pre-device selection rule is: in a preset learning cycle, the number of requests for the initial learning data packet is greater than the preset number of requests, the deduplication rate of the initial learning data packet meets the preset deduplication rule and the bandwidth of the initial learning data packet is greater than the second preset total bandwidth proportion threshold, wherein the second preset total bandwidth proportion threshold is less than the first preset total bandwidth proportion threshold.

[0062] Specifically, the preset learning period is to execute c times the time for which the deduplication buffer can store data, wherein c>b, optionally, 10≤c≤14, preferably, c=12.

[0063] Specifically, the preset deduplication rule is: the deduplication rate is greater than the second preset deduplication rate. Optionally, the value range of the second preset deduplication rate is 0.2 to 0.5; preferably, the second preset deduplication rate is 0.4. The strictness of the preset deduplication rule is higher than the traffic deduplication monitoring rule. It can be understood that the second preset deduplication rate is greater than the first preset deduplication rate.

[0064] S430: If there is a candidate triplet with a bandwidth greater than a first election triplet, replace the first election triplet with the candidate triplet to obtain a list of first election triplet after replacement. r The bandwidth is greater than the first election triple D sbandwidth, C r Replace D s .

[0065] S440, if the deduplication rate corresponding to the replaced first election triple list meets the preset deduplication rule, determine the replaced first election triple list as the final deduplication triple list; otherwise, execute the previous step S430.

[0066] In summary, a set of first election triple lists is obtained. If there is a first election triple list whose corresponding deduplication rate satisfies the preset deduplication rule, the first election triple list is determined as the final deduplication triple list; otherwise, an alternative triple list is obtained. If there is an alternative triple whose bandwidth is greater than that of a first election triple, the first election triple is replaced by the alternative triple to obtain the replaced first election triple list. If the corresponding deduplication rate of the replaced first election triple list satisfies the preset deduplication rule, the replaced first election triple list is determined as the final deduplication triple list; otherwise, replacement is continued until the corresponding deduplication rate of the replaced first election triple list satisfies the preset deduplication rule; by marking with the preset deduplication algorithm, the final deduplication triple is obtained using the alternative triple and the first election triple method, so that the deduplication data is more accurate and the transmission efficiency after deduplication is higher.

[0067] Specifically, Figure 3 As shown, in S400, the slice data packet corresponding to the final deduplication triplet is replaced with the mapping value of the slice data packet corresponding to the final deduplication triplet to obtain the deduplication target data for transmission, and further includes:

[0068] S401, obtaining a mapping value of a slice data packet corresponding to a final deduplication triplet as a final deduplication mapping value. In one embodiment of the present invention, the mapping value corresponding to the final deduplication triplet is a hash value corresponding to the final deduplication triplet.

[0069] S402, replacing the slice data packet corresponding to the final deduplication triplet with the final deduplication mapping value, packaging the replaced slice data packet list, and obtaining the deduplication target data.

[0070] S403, sending the deduplicated target data to the receiving end by replacing the slice data packet corresponding to the final triplet with the mapping value corresponding to the final deduplicated triplet to perform deduplication.

[0071] Furthermore, after the receiver receives the deduplicated target data, Figure 4 As shown, perform the following steps:

[0072] S404, parse the deduplicated target data to obtain the parsed final deduplication mapping value and the parsed slice data packet.

[0073] S405, obtaining a slice data packet corresponding to the final de-remapped value after parsing, and marking the slice data packet corresponding to the final de-remapped value after parsing as a parsed de-remapped data packet.

[0074] S406, reorganize the parsed deduplicated data packet and the parsed sliced ​​data packet to obtain reorganized data.

[0075] S407, verifying the reorganized data, if the verification is successful, the target data transmission is considered completed. The transmission accuracy of the target transmission is determined by parsing and reorganizing the final deduplicated data and obtaining the reorganized data for verification.

[0076] Furthermore, after performing the first detection algorithm to obtain the final deduplication triple list, the method further includes: performing detection using a second detection algorithm, wherein the second detection algorithm performs detection using the first detection algorithm once every preset detection time. Specifically, the preset detection time can be determined according to actual needs.

[0077] Furthermore, the present invention also includes: if the transmission bandwidth of the target data is less than the third preset total bandwidth ratio threshold value within the preset monitoring time period, stop using the final deduplication triplet to deduplicate the target data. Wherein, the third preset total bandwidth ratio threshold value is less than the first preset total bandwidth ratio threshold value.

[0078] Embodiment 2

[0079] Embodiment 2 of the present invention provides a flow deduplication device, such as Figure 5 As shown, the device comprises:

[0080] The target data acquisition module 10 is used to acquire the target data sent by the sender, and slice the target data to obtain a slice data packet list, wherein the slice data packet list includes a plurality of slice data packets.

[0081] The preset deduplication algorithm module 20 is used to obtain the transmission bandwidth of the target data. If the transmission bandwidth of the target data meets the preset total bandwidth ratio requirement, the slice data packet is identified based on the preset deduplication algorithm to obtain an initial deduplication data packet list and an initial learning data packet list, wherein the initial deduplication data packet list includes several initial deduplication data packets with the same identification, and the initial learning data packet list includes several initial learning data packets with the same identification, and the identification of the initial deduplication data packet is different from the identification of the initial learning data packet.

[0082] The network traffic deduplication monitoring module 30 is used to perform network traffic deduplication monitoring on target data within a preset time period and obtain network traffic deduplication monitoring results.

[0083] The first detection algorithm module 40 is used to obtain the final deduplication triplet list using the first detection algorithm if the network traffic deduplication monitoring result meets the traffic deduplication monitoring requirements, and deduplicate the target data based on the final deduplication triplet list to obtain the deduplication target data for transmission.

[0084] Among them, Figure 6 As shown, the first detection algorithm module 40 includes the following submodules:

[0085] The first election triplet acquisition submodule 41 is used to obtain a first election triplet list set, wherein the first election triplet list set includes several first election triplet lists, the first election triplet list includes several first election triplets, and the data packet corresponding to the first election triplet is an initial deduplicated data packet in the initial deduplicated data packet list.

[0086] The first connection submodule 42 is used to determine the first-selected triplet list as the final deduplicated triplet list if there is a first-selected triplet list whose corresponding deduplication rate satisfies the preset deduplication rule; otherwise, obtain an alternative triplet list, wherein the alternative triplet list contains several alternative triplets, and the alternative triplets are transmission triplets corresponding to the initial learning data packets that meet the pre-equipped selection rules.

[0087] The replacement submodule 43 is configured to, if there is a candidate triplet whose bandwidth is greater than the bandwidth of a first election triplet, replace the first election triplet with the candidate triplet to obtain a list of first election triplet after replacement.

[0088] The second connection submodule 44 is used to determine the replaced first election triple list as the final deduplicated triple list if the deduplication rate corresponding to the replaced first election triple list meets the preset deduplication rule; otherwise, execute the previous step.

[0089] Specifically, Figure 7 As shown, the first detection algorithm module 40 further includes a sender deduplication submodule 50, and the sender deduplication submodule 50 includes the following submodules:

[0090] The mapping value acquisition submodule 51 is used to obtain the mapping value of the slice data packet corresponding to the final deduplication triplet as the final deduplication mapping value.

[0091] The final deduplication data acquisition submodule 52 is used to replace the slice data packet corresponding to the final deduplication triplet with the final deduplication mapping value, package the replaced slice data packet list, and obtain the deduplication target data.

[0092] The sending submodule 53 is used to send the deduplicated target data to a receiver.

[0093] Specifically, Figure 8 As shown, the first detection algorithm module 40 further includes the following receiving-side deduplication submodule 60, and the receiving-side deduplication submodule 60 includes the following submodules:

[0094] The parsing submodule 61 is used to parse the deduplicated target data to obtain the parsed final deduplication mapping value and the parsed slice data packet.

[0095] The final de-duplication data packet acquisition submodule 62 is used to acquire the slice data packet corresponding to the parsed final de-duplication mapping value, and mark the slice data packet corresponding to the parsed final de-duplication mapping value as a parsed de-duplication data packet.

[0096] The reorganized data acquisition submodule 63 is used to reorganize the parsed deduplicated data packets and the parsed sliced ​​data packets to obtain reorganized data.

[0097] The target data transmission completion module 64 is used to verify the reorganized data. If the verification is successful, it is considered that the target data transmission is completed.

[0098] Embodiment 3

[0099] Embodiment 3 of the present invention provides a non-transitory computer-readable storage medium, wherein at least one instruction or at least one program is stored in the non-transitory computer-readable storage medium, and the at least one instruction or at least one program is loaded and executed by a processor to implement the steps:

[0100] S100, obtaining target data sent by a sender, and slicing the target data to obtain a slice data packet list, wherein the slice data packet list includes a plurality of slice data packets;

[0101] S200, obtaining a transmission bandwidth of the target data. If the transmission bandwidth of the target data does not meet the preset total bandwidth ratio requirement, the slice data packets are identified based on a preset deduplication algorithm, and an initial deduplication data packet list and an initial learning data packet list are obtained, wherein the initial deduplication data packet list includes a plurality of initial deduplication data packets with a first identification, and the initial learning data packet list includes a plurality of initial learning data packets with a second identification, wherein the first identification is different from the second identification;

[0102] S300, performing network traffic deduplication monitoring on target data in a preset time period to obtain a network traffic deduplication monitoring result;

[0103] S400, if the network traffic deduplication monitoring result meets the traffic deduplication monitoring rule, use the first detection algorithm to obtain the final deduplication triplet list, and use the mapping value of the slice data packet corresponding to the final deduplication triplet to replace the slice data packet corresponding to the final deduplication triplet, so as to obtain the deduplicated target data for transmission, and the final deduplication triplet list includes several final deduplication triples;

[0104] The first detection algorithm comprises the following steps:

[0105] S410, obtaining a first election triplet list, wherein the first election triplet list includes a plurality of first election triplets, and the first election triplet is a communication identifier corresponding to an initial deduplication data packet;

[0106] S420, if the deduplication rate corresponding to the first election triplet list meets the preset deduplication rule, the first election triplet list is determined as the final deduplication triplet list; otherwise, an alternative triplet list is obtained, wherein the alternative triplet list includes a plurality of alternative triplets, and the alternative triplets are communication identifiers corresponding to the initial learning data packets that meet the pre-equipped selection rule, and the preset deduplication rule is stricter than the traffic deduplication monitoring rule;

[0107] S430, if there is a candidate triplet with a bandwidth greater than a first election triplet, replace the first election triplet with the candidate triplet, and obtain a list of first election triplet after replacement;

[0108] S440, if the deduplication rate corresponding to the replaced first election triple list meets the preset deduplication rule, determine the replaced first election triple list as the final deduplication triple list; otherwise, execute the previous step.

[0109] An embodiment of the present invention further provides an electronic device, comprising a processor and the aforementioned non-transitory computer-readable storage medium.

[0110] Although some specific embodiments of the present invention have been described in detail by way of example, it should be understood by those skilled in the art that the above examples are only for illustration, not for limiting the scope of the present invention. It should also be understood by those skilled in the art that various modifications may be made to the embodiments without departing from the scope and spirit of the present invention.

Claims

1. A traffic deduplication method, the method is used to transmit target data from a sender to a receiver, characterized in that: The method includes the following contents: Obtain target data sent by a sender, and slice the target data to obtain a slice data packet list, wherein the slice data packet list includes a plurality of slice data packets; Obtaining the transmission bandwidth of the target data, if the transmission bandwidth of the target data does not meet the preset total bandwidth ratio requirement, identifying the slice data packets based on the preset deduplication algorithm, and obtaining an initial deduplication data packet list and an initial learning data packet list, wherein the initial deduplication data packet list includes a number of initial deduplication data packets with a first identifier, and the initial learning data packet list includes a number of initial learning data packets with a second identifier, wherein the first identifier is different from the second identifier; Perform network traffic deduplication monitoring on target data in a preset time period to obtain network traffic deduplication monitoring results; If the network traffic deduplication monitoring result meets the traffic deduplication monitoring rule, use the first detection algorithm to obtain the final deduplication triplet list, and use the mapping value of the slice data packet corresponding to the final deduplication triplet to replace the slice data packet corresponding to the final deduplication triplet, so as to obtain the deduplicated target data for transmission, and the final deduplication triplet list includes several final deduplication triplets; The first detection algorithm comprises the following steps: Obtaining a first election triplet list, wherein the first election triplet list includes a plurality of first election triplets, and the first election triplet is a communication identifier corresponding to an initial deduplication data packet; If the deduplication rate corresponding to the first election triplet list meets the preset deduplication rule, the first election triplet list is determined as the final deduplication triplet list; otherwise, an alternative triplet list is obtained, wherein the alternative triplet list contains a number of alternative triplets, and the alternative triplets are communication identifiers corresponding to the initial learning data packets that meet the pre-equipped selection rules, and the strictness of the preset deduplication rules is higher than the traffic deduplication monitoring rules, that is, the second preset deduplication rate is greater than the first preset deduplication rate; the preset deduplication rules are: the deduplication rate is greater than the second preset deduplication rate; if the network traffic deduplication monitoring result meets the traffic deduplication monitoring rules, the deduplication rate of the preset time period is greater than the first preset deduplication rate; If there is a candidate triplet with a bandwidth greater than a first election triplet, the first election triplet is replaced by the candidate triplet to obtain a list of first election triplet after replacement; If the deduplication rate corresponding to the replaced first election triple list meets the preset deduplication rule, the replaced first election triple list is determined as the final deduplication triple list; otherwise, the previous step is executed.

2. The traffic deduplication method according to claim 1, characterized in that: The detection size of the first detection algorithm is a preset multiple of the deduplication buffer size.

3. The traffic deduplication method according to claim 2, characterized in that: The mapping value of the slice data packet corresponding to the final deduplication triplet is used to replace the slice data packet corresponding to the final deduplication triplet to obtain the deduplication target data for transmission, and further includes: Obtaining the mapping value of the slice data packet corresponding to the final deduplication triplet as the final deduplication mapping value; The final deduplication mapping value is used to replace the slice data packet corresponding to the final deduplication triplet, and the replaced slice data packet list is packaged to obtain the deduplication target data; Send the deduplicated target data to the receiver.

4. The traffic deduplication method according to claim 3, characterized in that: After the receiver receives the deduplicated target data, it performs the following steps: Parse the deduplicated target data to obtain the final deduplicated mapping value and the parsed slice data packet; Obtaining a slice data packet corresponding to the final de-remapped value of the parsing, and marking the slice data packet corresponding to the final de-remapped value of the parsing as a parsed de-remapped data packet; Recombining the parsed deduplicated data packets and the parsed sliced ​​data packets to obtain recombined data; The reorganized data is verified. If the verification is successful, the target data transmission is considered completed.

5. The traffic deduplication method according to claim 1, characterized in that: The pre-device selection rule is: in a preset learning cycle, the number of requests for the initial learning data packet is greater than the preset number of requests, the deduplication rate of the initial learning data packet meets the preset deduplication rule and the bandwidth of the initial learning data packet is greater than the second preset total bandwidth proportion threshold, wherein the second preset total bandwidth proportion threshold is less than the preset bandwidth proportion threshold.

6. The traffic deduplication method according to claim 1, characterized in that: After performing the first detection algorithm to obtain the final deduplicated triple list, the method further includes: performing detection using a second detection algorithm, wherein the second detection algorithm performs detection of the first detection algorithm once at every preset detection time interval.

7. The traffic deduplication method according to claim 1, characterized in that: Also includes: If within the preset monitoring time period, the transmission bandwidth of the target data is less than the third preset total bandwidth ratio threshold, stop using the slice data packet corresponding to the final deduplication triplet to deduplicate the target data.

8. A flow deduplication device, characterized in that: The device comprises: A target data acquisition module is used to acquire target data sent by a sender, and slice the target data to acquire a slice data packet list, wherein the slice data packet list includes a plurality of slice data packets; A preset deduplication algorithm module is used to obtain the transmission bandwidth of the target data. If the transmission bandwidth of the target data does not meet the preset total bandwidth ratio requirement, the slice data packet is marked based on the preset deduplication algorithm to obtain an initial deduplication data packet list and an initial learning data packet list, wherein the initial deduplication data packet list includes a plurality of initial deduplication data packets with a first mark, and the initial learning data packet list includes a plurality of initial learning data packets with a second mark, wherein the first mark is different from the second mark; A network traffic deduplication monitoring module is used to perform network traffic deduplication monitoring on target data for a first preset time period and obtain network traffic deduplication monitoring results; A first detection algorithm module is used to obtain a final deduplication triplet list using the first detection algorithm if the network traffic deduplication monitoring result meets the traffic deduplication monitoring rule, and replace the slice data packet corresponding to the final deduplication triplet with the mapping value of the slice data packet corresponding to the final deduplication triplet to obtain the deduplicated target data for transmission, wherein the final deduplication triplet list includes several final deduplication triplets; The first detection algorithm module includes the following submodules: A first election triplet acquisition submodule, used to acquire a first election triplet list, wherein the first election triplet list includes a plurality of first election triplets, and the first election triplet is a communication identifier corresponding to an initial deduplication data packet; The first connection submodule is used to determine the first election triplet list as the final deduplication triplet list if the deduplication rate corresponding to the first election triplet list meets the preset deduplication rule; otherwise, obtain an alternative triplet list, wherein the alternative triplet list includes a plurality of alternative triplets, and the alternative triplets are communication identifiers corresponding to the initial learning data packets that meet the pre-equipped selection rule, and the strictness of the preset deduplication rule is higher than the traffic deduplication monitoring rule, that is, the second preset deduplication rate is greater than the first preset deduplication rate; the preset deduplication rule is: the deduplication rate is greater than the second preset deduplication rate; if the network traffic deduplication monitoring result meets the traffic deduplication monitoring rule, the deduplication rate of the preset time period is greater than the first preset deduplication rate; A replacement submodule, used for replacing the first election triplet with the candidate triplet if the bandwidth of the candidate triplet is greater than the bandwidth of the first election triplet, and obtaining a list of first election triplet after replacement; The second connection submodule is used to determine the replaced first election triple list as the final deduplicated triple list if the deduplication rate corresponding to the replaced first election triple list meets the preset deduplication rule; otherwise, execute the replacement submodule.

9. A non-transitory computer-readable storage medium, wherein at least one instruction or at least one program is stored in the storage medium, characterized in that: The at least one instruction or the at least one program is loaded and executed by the processor to implement the traffic deduplication method as described in any one of claims 1-7.

10. An electronic device, characterized in that: The invention comprises a processor and the non-transitory computer-readable storage medium as claimed in claim 9.

Images