Traffic de-duplication method, device and equipment and readable storage medium
By analyzing and deduplication of the network traffic of the test subject, generating structure definition schema and verification mark, the traffic redundancy problem in the test is solved and the testing efficiency is improved.
Patent Information
- Application Number
- CN202411741693.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-29
- Publication Date
- 2025-02-25
AI Technical Summary
During the test, there is a large amount of duplication and redundant traffic in the traffic recording, resulting in inefficiency in the test.
By capturing the network traffic of the target test subject, analyzing and extracting the incoming and outgoing parameters, a structure definition schema of the network traffic is generated, and the schema is processed to generate a verification mark, and the verification mark is compared to deduplicate similar traffic.
By removing redundant network traffic, the difficulty of traffic comparison is simplified, the testing efficiency is improved, and the omissions of test scenarios are avoided.
Smart Images
Figure CN119520325A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of testing technology, and more specifically, to a traffic deduplication method, device, equipment and readable storage medium. Background Art
[0002] After preparing a new test subject, the correctness of the test subject code logic can be verified through traffic recording and traffic playback. In this process, traffic recording may record a lot of repeated traffic. The redundant traffic makes it difficult for testers to quickly understand the situation of the test subject, resulting in low test efficiency.
[0003] On this basis, how to deduplicate traffic to improve test efficiency has become the focus of researchers in this field. Summary of the invention
[0004] In view of this, the present application provides a traffic deduplication method, device, equipment and readable storage medium to solve the shortcomings of low testing efficiency in the prior art.
[0005] In order to achieve the above objectives, the proposed solution is as follows:
[0006] A traffic deduplication method, comprising:
[0007] Capture multiple network flows of the target test subject;
[0008] For each network flow, parse the network flow and extract the input parameter data and output parameter data of the network flow; generate a structure definition schema of the network flow based on the input parameter data and the output parameter data; process the schema of the network flow and generate a verification mark for describing the information contained in the schema;
[0009] The verification identifiers of each network flow are compared to remove duplicate network flows corresponding to the same verification identifier.
[0010] Optionally, generating a structure definition schema of the network traffic based on the input parameter data and the output parameter data includes:
[0011] Utilizing a data pattern inference tool, the input parameter data is parsed to generate an input parameter structure definition;
[0012] Utilizing the data pattern inference tool, parsing the output parameter data to generate an output parameter structure definition;
[0013] The input parameter structure definition and the output parameter structure definition are integrated to generate a structure definition schema of the network traffic.
[0014] Optionally, processing the schema of the network traffic to generate a verification identifier for describing information included in the schema includes:
[0015] Obfuscating personalized key names in the schema of the network traffic;
[0016] Generates a verification token based on the obfuscated schema.
[0017] Optionally, the personalized key name in the schema of the network traffic is fuzzified, including:
[0018] Identify the privacy key names in the schema and perform fuzzification on all privacy key names.
[0019] Optionally, the identifying the privacy key names in the schema and performing fuzzy processing on all privacy key names includes:
[0020] Identify privacy keys in the schema and determine the key type of each privacy key;
[0021] Replace each privacy key name with the preset obfuscated characters corresponding to the key name type.
[0022] Optionally, generating a verification mark based on the fuzzified schema includes:
[0023] The obfuscated schema is signed, and a verification mark is obtained after signing.
[0024] Optionally, generating a verification mark based on the fuzzified schema includes:
[0025] A hash algorithm is used to calculate a hash value of the obfuscated schema, where the hash value is the verification identifier.
[0026] A flow deduplication device, comprising:
[0027] A capture module, used to capture multiple network flows of a target test subject;
[0028] A generation module is used to analyze the network flow for each network flow, and extract the input parameter data and output parameter data of the network flow; generate a structure definition schema of the network flow based on the input parameter data and the output parameter data; process the schema of the network flow, and generate a verification mark for describing the information contained in the schema;
[0029] The deduplication module is used to compare the verification identifiers of various network flows and to dedupe the network flows corresponding to the same verification identifier.
[0030] A traffic deduplication device, comprising a memory and a processor;
[0031] The memory is used to store programs;
[0032] The processor is used to execute the program to implement each step of the above-mentioned traffic deduplication method.
[0033] A readable storage medium having a computer program stored thereon, characterized in that when the computer program is executed by a processor, each step of the above-mentioned traffic deduplication method is implemented.
[0034] It can be seen from the above technical solutions that the traffic deduplication method provided by the present application can capture multiple network flows of the target test subject; for each network flow, the network flow is parsed, and the input parameter data and output parameter data of the network flow are extracted; based on the input parameter data and the output parameter data, a structure definition schema of the network flow is generated; the schema of the network flow is processed to generate a verification mark for describing the information contained in the schema; based on this, the present application can analyze the input parameters and output parameters of each network flow, construct the schema of each network flow, and generate a verification mark for each schema; since the schema is a declarative format for describing other data structures, the schema can clearly reflect the data elements contained in the input parameters and output parameters of each network flow, and clearly reflect the data types contained in the input parameters and output parameters of each network flow. Convert the network traffic with huge data volume into schema, extract important information, and remove redundant information; after generating the check mark of the schema, the check mark can be used to reflect the data elements, data types and dependencies contained in the schema, and further compress the data volume of the network traffic; then, the check marks of each network traffic can be compared to deduplicate the network traffic corresponding to the same check mark; based on this, the present application can identify similar network traffic by comparing the check marks, thereby deduplicating each network traffic. It can be seen that the present application can convert the comparison between multiple network traffic into the comparison between multiple check marks, and by generating schema and check marks, the data volume of network traffic can be doubly compressed, and the difficulty of network traffic comparison can be simplified, thereby improving the test efficiency and avoiding the omission of test scenarios. BRIEF DESCRIPTION OF THE DRAWINGS
[0035] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.
[0036] Figure 1 A flow chart of a traffic deduplication method disclosed in an embodiment of the present application;
[0037] Figure 2 A structural block diagram of a traffic deduplication device disclosed in an embodiment of the present application;
[0038] Figure 3 This is a hardware structure block diagram of a traffic deduplication device disclosed in an embodiment of the present application. DETAILED DESCRIPTION
[0039] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0040] The traffic deduplication method of the present application can be used in many general or special computing device environments or configurations, such as personal computers, server computers, handheld devices or portable devices, tablet devices, multi-processor devices, and distributed computing environments including any of the above devices or equipment.
[0041] Next, we will combine Figure 1 The traffic deduplication method of this application is introduced in detail, including the following steps:
[0042] Step S1: Capture multiple network flows of the target test subject.
[0043] Specifically, the target test subject may be different types of objects that need to be tested.
[0044] For example, the target test subject may be a newly constructed business service object or an updated business service object.
[0045] The business service object may be a variety of different types of objects, for example, an interface, a program module, and an app.
[0046] There are many ways to capture multiple network flows of the target test subject.
[0047] For example, you can use network packet sniffing tools such as Wireshark and Tcpdump to capture multiple network flows of the target test subject.
[0048] You can also use the traffic replication function of the router to capture multiple network flows of the target test subject.
[0049] Step S2: for each network flow, parse the network flow and extract the input parameter data and output parameter data of the network flow; based on the input parameter data and the output parameter data, generate a structure definition schema of the network flow; process the schema of the network flow and generate a verification identifier for describing the information contained in the schema.
[0050] Specifically, each captured network flow can be parsed one by one, and input parameter related data can be extracted from each network flow as the input parameter data of the corresponding network flow, and output parameter related data can be extracted from each network flow as the output parameter data of the corresponding network flow.
[0051] The input data may be used to determine the input requirements of the target test subject, such as input data elements and input data types.
[0052] Similarly, the input data can be used to determine the response result of the target test subject, such as the output data elements and output data types.
[0053] The data types and structural relationships between the data elements contained in the input parameter data and the output parameter data can be analyzed to generate a structural definition schema of the network traffic;
[0054] Some key names in the schema of the network traffic may be fuzzified, and based on the fuzzified schema, a verification identifier for describing information included in the schema may be generated.
[0055] Step S3: compare the verification identifiers of each network flow, and remove duplicate network flows corresponding to the same verification identifier.
[0056] Specifically, the similarity between the verification identifier of each network flow and the verification identifiers of other network flows may be calculated;
[0057] De-duplicate the network traffic corresponding to the verification identifiers whose similarity exceeds the preset similarity threshold.
[0058] There are many ways to calculate the similarity between the verification identifier of each network flow and the verification identifiers of other network flows.
[0059] For example, the Euclidean distance between the check mark of each network flow and the check marks of other network flows can be calculated;
[0060] The cosine similarity between the verification mark of each network flow and the verification marks of other network flows can also be calculated;
[0061] It is also possible to calculate the Jaccard similarity between the verification mark of each network flow and the verification marks of other network flows;
[0062] The similarity between the verification mark of each network flow and the verification marks of other network flows can also be calculated based on the word vector averaging method.
[0063] It can be seen from the above technical solutions that the traffic deduplication method provided in the embodiment of the present application can capture multiple network flows of the target test subject; for each network flow, parse the network flow, and extract the input parameter data and output parameter data of the network flow; based on the input parameter data and the output parameter data, generate a structure definition schema of the network flow; process the schema of the network flow to generate a verification mark for describing the information contained in the schema; based on this, the present application can analyze the input parameters and output parameters of each network flow, construct the schema of each network flow, and generate a verification mark for each schema; since the schema is a declarative format for describing other data structures, the schema can clearly reflect the data elements contained in the input parameters and output parameters of each network flow, and clearly reflect the data types contained in the input parameters and output parameters of each network flow. Convert the network traffic with huge data volume into schema, extract important information, and remove redundant information; after generating the check mark of the schema, the check mark can be used to reflect the data elements, data types and dependencies contained in the schema, and further compress the data volume of the network traffic; then, the check marks of each network traffic can be compared to deduplicate the network traffic corresponding to the same check mark; based on this, the present application can identify similar network traffic by comparing the check marks, thereby deduplicating each network traffic. It can be seen that the present application can convert the comparison between multiple network traffic into the comparison between multiple check marks, and by generating schema and check marks, the data volume of network traffic can be doubly compressed, and the difficulty of network traffic comparison can be simplified, thereby improving the test efficiency and avoiding the omission of test scenarios.
[0064] In some embodiments of the present application, the process of generating the structure definition schema of the network traffic based on the input parameter data and the output parameter data in step S2 is described in detail, and the steps are as follows:
[0065] S20. Utilize a data pattern inference tool to parse the input parameter data and generate an input parameter structure definition.
[0066] Specifically, the open source framework json-schema-inferrer in github can be used to parse the input data elements in the input data, the input data types corresponding to each input data element, and the structural relationship between each input data element to generate an input structure definition.
[0067] S21. Utilize the data pattern inference tool to parse the output parameter data and generate an output parameter structure definition.
[0068] Specifically, the open source framework json-schema-inferrer in github can be used to parse the output parameter data elements in the parameter data, the output parameter data types corresponding to each output parameter data element, and the structural relationship between each output parameter data element to generate the output parameter structure definition.
[0069] S23. Integrate the input parameter structure definition and the output parameter structure definition to generate a structure definition schema of the network traffic.
[0070] Specifically, the structural relationship between the input parameter data elements and the output parameter data elements in the input parameter data and the output parameter data can be analyzed, the input parameter structure definition and the output parameter structure definition can be integrated, and the structural definition schema of the network traffic can be generated.
[0071] It can be seen from the above technical solution that this embodiment provides an optional method for generating the structural definition schema of the network traffic based on the input parameter data and the output parameter data. Through the above method, the input parameter data elements, input parameter data types and structural relationships between the input parameter data elements in the input parameter data, and the output parameter data elements, output parameter data types and structural relationships between the output parameter data elements in the output parameter data can be integrated to generate the structural definition schema of the network traffic, thereby better extracting important information of the network traffic and constructing the schema.
[0072] In some embodiments of the present application, the process of processing the schema of the network traffic in step S2 to generate a verification identifier for describing the information included in the schema is described in detail, and the steps are as follows:
[0073] S24. Perform fuzzy processing on the personalized key names in the schema of the network traffic.
[0074] Specifically, since similar test results are obtained when network traffic in similar scenarios is used for testing, the network traffic in similar scenarios needs to be reduced;
[0075] In order to reduce the traffic corresponding to similar scenarios, you can try to deduplicate the network traffic of similar scenarios.
[0076] A unified fuzzification format can be used to fuzzify personalized key names in the schema of network traffic to achieve consistency in the fuzzified schema of similar scenarios.
[0077] Among them, the fuzzification formats corresponding to different target test subjects are different, and the fuzzification format can be set according to actual needs.
[0078] Similarly, different target test subjects require different fuzzy key names, and personalized key names that need to be fuzzy can be set according to actual needs.
[0079] For example, when the target test subject is a shopping software, the personalized key name that needs to be obfuscated may be an order number, a user ID, a product ID, and the like.
[0080] For another example, when the target test subject is information interaction software, the personalized key name that needs to be obfuscated may be a user ID, an interaction information ID, and the like.
[0081] On this basis, the type of personalized key name that needs to be fuzzified can be determined based on the type of the target test subject;
[0082] Identify whether each key name in the network traffic schema corresponds to each personalized key name type of the target test subject;
[0083] If yes, then fuzzify the corresponding key name;
[0084] If not, keep the corresponding key name.
[0085] S25. Generate a verification mark based on the fuzzified schema.
[0086] Specifically, the fuzzified schema can be processed in a variety of ways to obtain a verification mark.
[0087] For example, a signature algorithm can be used to sign the obfuscated schema, and a verification mark can be obtained after signing;
[0088] The information summary algorithm can be used to process the obfuscated schema to obtain a verification mark.
[0089] The message authentication code algorithm MAC can be used to process the obfuscated schema and obtain a verification mark after processing;
[0090] You can also train an open source neural network to obtain a verification mark generation model;
[0091] The fuzzified schema is input into the verification mark generation model to obtain the verification mark corresponding to the fuzzified schema.
[0092] It can be seen from the above technical solution that this embodiment provides an optional method for processing the schema of the network traffic and generating a verification identifier for describing the information contained in the schema. Through the above method, the personalized key name of the schema can be fuzzified, and then a verification identifier can be generated based on the fuzzified schema, so as to realize the conversion of schemas of similar scenarios into the same verification identifier, so as to identify network traffic of similar scenarios in subsequent verification identifier comparisons, thereby further removing redundant network traffic and improving test efficiency.
[0093] In some embodiments of the present application, the process of performing fuzzy processing on the personalized key name in the schema of the network traffic in step S24 is described in detail, and the steps are as follows:
[0094] S240: Identify the privacy key names in the schema, and perform fuzzification processing on all privacy key names.
[0095] Specifically, each key name in the schema can be parsed to identify whether each key name is a privacy key name;
[0096] If yes, the privacy key name is obfuscated;
[0097] If not, keep the corresponding key name.
[0098] The privacy key name can be various identifiers that cannot be disclosed, for example, a user identifier, an order identifier, and interaction information.
[0099] It can be seen from the above technical solution that this embodiment provides an optional method for fuzzifying the personalized key names in the schema of the network traffic. Through the above method, the present application can further fuzzify the privacy key names, thereby ensuring the security of the network traffic, and further ensuring the security and reliability of the test.
[0100] In some embodiments of the present application, the process of S240, identifying the privacy key names in the schema and performing fuzzy processing on all privacy key names is described in detail, and the steps are as follows:
[0101] S2400: Identify the privacy key names in the schema, and determine the key name type of each privacy key name.
[0102] Specifically, it can identify all key names in the schema and the key name type of each key name;
[0103] According to the key name type of each key name, determine whether the corresponding key name is a privacy key name.
[0104] S2401. Replace each privacy key name with a preset fuzzy character corresponding to the key name type.
[0105] Specifically, the fuzzy characters and / or fuzzification method corresponding to the key name type of each privacy key name may be set;
[0106] Among them, the key name types of different privacy key names can correspond to the same obfuscated characters and / or obfuscation methods.
[0107] Based on the key name type of each privacy key name, the corresponding privacy key name is replaced with the corresponding obfuscated characters and / or obfuscation method.
[0108] It can be seen from the above technical solution that this embodiment provides an optional method for identifying the privacy key names in the schema and fuzzifying all privacy key names. The above method can better fuzzify the privacy key names, thereby better deduplicating the various network traffic of the target test subject.
[0109] In some embodiments of the present application, two methods of executing step S25 and generating a verification mark based on the fuzzified schema are provided. Next, the above two execution methods will be described in detail as follows:
[0110] The first one,
[0111] S250: Sign the obfuscated schema, and obtain a verification mark after signing.
[0112] Specifically, the obfuscated schema may be signed using an RSA signature algorithm, a DSA (Digital Signature Algorithm) signature algorithm, an ECDSA (Elliptic Curve Digital Signature Algorithm) signature algorithm, or an HMAC (Hash-based Message Authentication Code) signature algorithm, and a verification mark may be obtained after signing.
[0113] The second type
[0114] S251. Calculate the hash value of the obfuscated schema using a hash algorithm, where the hash value is the verification identifier.
[0115] Specifically, the hash value of the obfuscated schema may be calculated using an MD5 (Message Digest Algorithm 5) hash algorithm, a SHA-1 (Secure Hash Algorithm 1) hash algorithm, a SHA-2 (Secure Hash Algorithm 2) hash algorithm, or a CRC (Cyclic Redundancy Check) hash algorithm, and the hash value is the verification identifier.
[0116] It should be noted that the schema of the network traffic corresponding to the same target test subject uses the same algorithm to generate verification identifiers.
[0117] It can be seen from the above technical solution that this embodiment provides two optional methods for generating verification marks based on the fuzzy schema. Through the above methods, the present application can select different verification mark generation methods according to different scenarios, thereby further improving the reliability of the present application.
[0118] Next, we will combine Figure 2 The traffic deduplication device provided in the present application is introduced in detail. The traffic deduplication device provided below can be compared with the traffic deduplication method provided above.
[0119] See also Figure 2 It can be found that the traffic deduplication device may include:
[0120] A capture module 10, used to capture multiple network flows of a target test subject;
[0121] The generating module 20 is used to analyze the network flow for each network flow, and extract the input parameter data and the output parameter data of the network flow; generate the structure definition schema of the network flow based on the input parameter data and the output parameter data; process the schema of the network flow, and generate a verification mark for describing the information contained in the schema;
[0122] The deduplication module 30 is used to compare the verification identifiers of various network flows and to dedupe the network flows corresponding to the same verification identifier.
[0123] Furthermore, the generation module may include:
[0124] An input parameter structure definition generating unit, used to parse the input parameter data using a data pattern inference tool to generate an input parameter structure definition;
[0125] An output parameter structure definition generating unit, used to parse the output parameter data using the data pattern inference tool to generate an output parameter structure definition;
[0126] The schema generation unit is used to integrate the input parameter structure definition and the output parameter structure definition to generate a structure definition schema of the network traffic.
[0127] Furthermore, the generation module may also include:
[0128] A key name processing unit, used for performing fuzzy processing on personalized key names in the schema of the network traffic;
[0129] The verification mark generation unit is used to generate a verification mark based on the fuzzified schema.
[0130] Furthermore, the key name processing unit may include:
[0131] The privacy key name identification subunit is used to identify the privacy key names in the schema and perform fuzzy processing on all privacy key names.
[0132] Further, the privacy key name identification subunit may include:
[0133] A first privacy key name identification component, used to identify the privacy key names in the schema and determine the key name type of each privacy key name;
[0134] The second privacy key name recognition component is used to replace each privacy key name with a preset fuzzy character corresponding to the key name type.
[0135] Furthermore, the verification mark generation unit may include:
[0136] The schema signature subunit is used to sign the obfuscated schema and obtain a verification mark after signing.
[0137] Furthermore, the verification mark generation unit may further include:
[0138] The hash value calculation subunit is used to calculate the hash value of the obfuscated schema by using a hash algorithm, and the hash value is the verification identifier.
[0139] The traffic deduplication device provided in the embodiment of the present application can be applied to traffic deduplication equipment, such as PC terminals, cloud platforms, servers and server clusters. Figure 3 The hardware structure diagram of the traffic deduplication device is shown in FIG. Figure 3 ,The hardware structure of the traffic deduplication device may include: at least one processor 1, at least one communication interface 2, at least one memory 3, and at least one communication bus 4;
[0140] In the embodiment of the present application, the number of the processor 1, the communication interface 2, the memory 3, and the communication bus 4 is at least one, and the processor 1, the communication interface 2, and the memory 3 communicate with each other through the communication bus 4;
[0141] The processor 1 may be a central processing unit CPU, or an application-specific integrated circuit ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of the present invention, etc.;
[0142] The memory 3 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), etc., such as at least one disk memory;
[0143] The memory stores a program, and the processor can call the program stored in the memory, wherein the program is used to:
[0144] Capture multiple network flows of the target test subject;
[0145] For each network flow, parse the network flow and extract the input parameter data and output parameter data of the network flow; generate a structure definition schema of the network flow based on the input parameter data and the output parameter data; process the schema of the network flow and generate a verification mark for describing the information contained in the schema;
[0146] The verification identifiers of each network flow are compared to remove duplicate network flows corresponding to the same verification identifier.
[0147] Optionally, the detailed functions and extended functions of the program may refer to the above description.
[0148] The embodiment of the present application further provides a readable storage medium, which may store a program suitable for execution by a processor, wherein the program is used to:
[0149] Capture multiple network flows of the target test subject;
[0150] For each network flow, parse the network flow and extract the input parameter data and output parameter data of the network flow; generate a structure definition schema of the network flow based on the input parameter data and the output parameter data; process the schema of the network flow and generate a verification mark for describing the information contained in the schema;
[0151] The verification identifiers of each network flow are compared to remove duplicate network flows corresponding to the same verification identifier.
[0152] Optionally, the detailed functions and extended functions of the program may refer to the above description.
[0153] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.
[0154] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.
[0155] The above description of the disclosed embodiments enables professionals and technicians in the field to implement or use the present application. Various modifications to these embodiments will be apparent to professionals and technicians in the field, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. The various embodiments of the present application can be combined with each other. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A traffic deduplication method, characterized in that: include: Capture multiple network flows of the target test subject; For each network flow, parse the network flow and extract the input parameter data and output parameter data of the network flow; Generate a structure definition schema of the network traffic based on the input parameter data and the output parameter data; Processing the schema of the network traffic to generate a verification identifier for describing information included in the schema; The verification identifiers of each network flow are compared to remove duplicate network flows corresponding to the same verification identifier.
2. The traffic deduplication method according to claim 1, characterized in that: The generating a structure definition schema of the network traffic based on the input parameter data and the output parameter data includes: Utilizing a data pattern inference tool, the input parameter data is parsed to generate an input parameter structure definition; Utilizing the data pattern inference tool, parsing the output parameter data to generate an output parameter structure definition; The input parameter structure definition and the output parameter structure definition are integrated to generate a structure definition schema of the network traffic.
3. The traffic deduplication method according to claim 1, characterized in that: Processing the schema of the network traffic to generate a verification identifier for describing information included in the schema includes: Obfuscating personalized key names in the schema of the network traffic; Generates a verification token based on the obfuscated schema.
4. The traffic deduplication method according to claim 3, characterized in that: Performing fuzzy processing on the personalized key names in the schema of the network traffic, including: Identify the privacy key names in the schema and perform fuzzification on all privacy key names.
5. The traffic deduplication method according to claim 4, characterized in that: The identifying the privacy key names in the schema and performing fuzzy processing on all privacy key names include: Identify privacy key names in the schema and determine the key name type of each privacy key name; Replace each privacy key name with the preset obfuscated characters corresponding to the key name type.
6. The traffic deduplication method according to claim 3, characterized in that: The generating of the verification mark based on the fuzzified schema includes: The obfuscated schema is signed, and a verification mark is obtained after signing.
7. The traffic deduplication method according to claim 3, characterized in that: The generating of the verification mark based on the fuzzified schema includes: A hash algorithm is used to calculate a hash value of the obfuscated schema, where the hash value is the verification identifier.
8. A flow deduplication device, characterized in that: include: A capture module, used to capture multiple network flows of a target test subject; A generation module is used to analyze the network traffic for each network traffic, and extract the input parameter data and output parameter data of the network traffic; Generate a structure definition schema of the network traffic based on the input parameter data and the output parameter data; Processing the schema of the network traffic to generate a verification identifier for describing information included in the schema; The deduplication module is used to compare the verification identifiers of various network flows and to dedupe the network flows corresponding to the same verification identifier.
9. A traffic deduplication device, characterized in that: including memory and processor; The memory is used to store programs; The processor is used to execute the program to implement each step of the traffic deduplication method as described in any one of claims 1-7.
10. A readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, each step of the traffic deduplication method as described in any one of claims 1 to 7 is implemented.