Traffic deduplication method, device, electronic device and storage medium

Abstract

The present application provides a traffic deduplication method, device, electronic device and storage medium. The method includes: obtaining traffic data to be deduplicated, the traffic data to be deduplicated includes multiple traffic data; wherein each of the traffic data includes a first identifier, a second identifier and recorded traffic, the first identifier is generated according to a recording interface corresponding to the recorded traffic, and the second identifier is generated according to a sub-call set and a service label corresponding to the recorded traffic; deduplication of the traffic data to be deduplicated is performed according to the first identifier and the second identifier corresponding to each traffic data. The present application improves the efficiency and accuracy of traffic deduplication by using the first identifier corresponding to the recording interface and the second identifier generated by the service label and the sub-call set to perform traffic deduplication.

Description

Technical Field

[0001] The present application relates to the field of software development and testing technology, and in particular to a traffic deduplication method, device, electronic device and storage medium. Background Art

[0002] After the business service is updated, it is necessary to test the business service through traffic recording and playback. During the test, the traffic data recorded in the live network is compared with the playback traffic data in the playback environment to generate the test results. Because there will be a large amount of repeated traffic in the recorded traffic, the test efficiency is low.

[0003] To solve this problem, currently, the recorded traffic is deduplicated by manually labeling the traffic, but this traffic deduplication method is inefficient. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a traffic deduplication method, device, electronic device and storage medium to improve the efficiency of traffic deduplication.

[0005] In a first aspect, an embodiment of the present application provides a traffic deduplication method, comprising:

[0006] Obtaining traffic data to be deduplicated, the traffic data to be deduplicated comprising multiple pieces of traffic data; wherein each piece of the traffic data comprises a first identifier, a second identifier and recorded traffic, the first identifier being generated according to a recording interface corresponding to the recorded traffic, and the second identifier being generated according to a subcall set and a service label corresponding to the recorded traffic;

[0007] The to-be-deduplicated traffic data is deduplicated according to the first identifier and the second identifier corresponding to each piece of traffic data.

[0008] The embodiment of the present application deduplicates traffic by utilizing a first identifier corresponding to a recording interface and a second identifier generated by a service tag and a sub-call set, thereby improving the efficiency and accuracy of traffic deduplication.

[0009] In any embodiment, deduplicating the to-be-deduplicated traffic data according to the first identifier and the second identifier corresponding to each piece of traffic data includes:

[0010] Determine the same traffic data in the recorded traffic according to the first identifier and the second identifier;

[0011] The identical traffic data is partially removed, and only one piece of traffic data is retained, so as to achieve deduplication of the traffic data to be deduplicated.

[0012] In the embodiment of the present application, the first identifier can be used to characterize the uniqueness of the recording interface, and the second identifier can be used to characterize the uniqueness of the sub-call. Therefore, deduplication of the recording traffic is performed using the first identifier and the second identifier, which can improve the accuracy of deduplication.

[0013] In any embodiment, before obtaining the traffic data to be deduplicated, the method further includes:

[0014] Acquire recorded traffic through a jvm sandbox repeater tool, where the recorded traffic includes a subcall set;

[0015] Extracting a recording interface of the recorded traffic and generating a first identifier according to the recording interface;

[0016] Generate a service tag corresponding to the recorded traffic, and generate a second identifier according to the service tag and the sub-call set.

[0017] The embodiment of the present application deduplicates traffic by utilizing a first identifier corresponding to a recording interface and a second identifier generated by a service tag and a sub-call set, thereby improving the efficiency and accuracy of traffic deduplication.

[0018] In any embodiment, the sub-call set includes at least one sub-call; and the call type corresponding to the at least one sub-call is determined in advance by the business server according to the hash value of the thread corresponding to the sub-call; wherein the call type includes a main thread sub-call and an asynchronous thread sub-call.

[0019] The embodiment of the present application can avoid many unnecessary scenarios and improve the accuracy of deduplication of recorded traffic by dividing the sub-calls into main thread sub-calls and asynchronous thread sub-calls.

[0020] In any embodiment, generating a second identifier according to the service tag and the sub-call set of recorded traffic includes:

[0021] Generate a main thread sub-call set for the sub-call of the main thread sub-call according to the call type;

[0022] Generate an asynchronous thread sub-call set according to the sub-call whose call type is an asynchronous thread sub-call;

[0023] Generate a target string from the main thread sub-call set, the asynchronous thread sub-call set and the business label according to a preset format;

[0024] The target character string is calculated using a preset algorithm to obtain the second identifier.

[0025] The embodiment of the present application can avoid many unnecessary scenarios by dividing the sub-calls into main thread sub-calls and asynchronous thread sub-calls, and combined with business tags, improves the accuracy of deduplication of recorded traffic.

[0026] In any embodiment, after generating the second identifier, the method further includes:

[0027] The first identifier, the second identifier and the traffic data corresponding to each piece of traffic data are stored in the search server as a piece of traffic data;

[0028] Accordingly, the step of obtaining the traffic data to be deduplicated includes:

[0029] The to-be-deduplicated traffic data within a preset time period is obtained from the search server.

[0030] Since the amount of recorded traffic data is large, in order to reduce the pressure on the recording platform, the preprocessed traffic data is stored in the search server, and then the data of the recorded traffic to be deduplicated is read from the search server to achieve the purpose of reducing the pressure on the recording platform.

[0031] In any embodiment, generating a service tag corresponding to the recorded traffic includes:

[0032] Extract preset fields from each flow data;

[0033] A business tag matching the preset field is determined from the pre-stored field tag correspondence.

[0034] The embodiment of the present application generates a corresponding service tag for each piece of traffic data, and uses the service tag to reflect whether the scenes corresponding to the traffic data are the same, so as to improve the accuracy of deduplication of recorded traffic.

[0035] In a second aspect, an embodiment of the present application provides a traffic deduplication device, comprising:

[0036] A data acquisition module, used to acquire the traffic data to be deduplicated, wherein the traffic data to be deduplicated includes multiple traffic data; wherein each of the traffic data includes a first identifier, a second identifier and recorded traffic, wherein the first identifier is generated according to the recording interface corresponding to the recorded traffic, and the second identifier is generated according to the subcall set and service label corresponding to the recorded traffic;

[0037] The deduplication module is used to deduplicate the traffic data to be deduplicated according to the first identifier and the second identifier corresponding to each piece of traffic data.

[0038] In a third aspect, an embodiment of the present application provides an electronic device, including: a processor, a memory, and a bus, wherein:

[0039] The processor and the memory communicate with each other via the bus;

[0040] The memory stores program instructions that can be executed by the processor, and the processor can execute the method of the first aspect by calling the program instructions.

[0041] In a fourth aspect, an embodiment of the present application provides a non-transitory computer-readable storage medium, including:

[0042] The non-transitory computer-readable storage medium stores computer instructions that cause the computer to execute the method of the first aspect.

[0043] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or be understood by practicing the embodiments of the present application. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0045] Figure 1 A flow chart of a traffic deduplication method provided in an embodiment of the present application;

[0046] Figure 2 A schematic diagram of another flow deduplication method provided in an embodiment of the present application;

[0047] Figure 3 A schematic diagram of the structure of a flow deduplication device provided in an embodiment of the present application;

[0048] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0049] The following embodiments of the technical solution of the present application are described in detail in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present application, and are therefore only used as examples, and cannot be used to limit the scope of protection of the present application.

[0050] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by technicians in the technical field to which this application belongs; the terms used herein are only for the purpose of describing specific embodiments and are not intended to limit this application; the terms "including" and "having" in the specification and claims of this application and the above-mentioned figure descriptions and any variations thereof are intended to cover non-exclusive inclusions.

[0051] In the description of the embodiments of the present application, the technical terms "first", "second", etc. are only used to distinguish different objects, and cannot be understood as indicating or implying relative importance or implicitly indicating the number, specific order or primary and secondary relationship of the indicated technical features. In the description of the embodiments of the present application, the meaning of "multiple" is more than two, unless otherwise clearly and specifically defined.

[0052] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0053] In the description of the embodiments of the present application, the term "and / or" is only a description of the association relationship of the associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0054] In the description of the embodiments of the present application, the term "multiple" refers to more than two (including two). Similarly, "multiple groups" refers to more than two groups (including two groups), and "multiple pieces" refers to more than two pieces (including two pieces).

[0055] In the description of the embodiments of the present application, the technical terms "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", "clockwise", "counterclockwise", "axial", "radial", "circumferential", etc. indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, which are only for the convenience of describing the embodiments of the present application and simplifying the description, and do not indicate or imply that the referred device or element must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the embodiments of the present application.

[0056] In the description of the embodiments of the present application, unless otherwise clearly specified and limited, technical terms such as "installed", "connected", "connected", "fixed" and the like should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, and it can be the internal connection of two elements or the interaction relationship between two elements. For those of ordinary skill in the art, the specific meanings of the above terms in the embodiments of the present application can be understood according to the specific circumstances.

[0057] The traffic generated by the production environment every day is massive and contains a large amount of duplication. A large number of repeated business requests cannot effectively test the business system to be tested. Therefore, a solution is needed to deduplicate a large number of system business requests in the production environment.

[0058] Currently, traffic can be controlled by manually labeling traffic data, that is, manually setting corresponding labels for each flow, removing the traffic with the same label, so that only one flow with the same label is retained, and deduplication of traffic is achieved. However, in the face of a large amount of traffic, the manual labeling method will result in a slow deduplication efficiency.

[0059] In order to solve the above technical problems, the inventor of the present application provides a traffic deduplication method, which distinguishes the differences and similarities of traffic data based on the recording interface, the service label corresponding to each traffic data, and the sub-call set corresponding to each traffic data, so as to achieve the purpose of deduplication of recorded traffic.

[0060] Before introducing the specific solution of this application, in order to better understand this application, the relevant concepts involved are introduced:

[0061] Entry call: The call formed by the request initiated by the outside is called the entry call. Currently, the traffic playback platform supports types such as http, dubbo, and mq (rocketMQ).

[0062] Subcall: A subcall formed by an out-of-process or Java (enhancement required) method, such as an out-of-process call (redis, mybatis), Java method enhancement, custom subcall, etc. Among them, subcalls can be divided into main thread subcalls and asynchronous thread subcalls.

[0063] JVM Sandbox Repeater: is a general solution for server-side recording / playback based on JVM-Sandbox.

[0064] Recording: The process of serializing and storing the input parameters, output parameters, downstream RPC, DB, cache, etc. of a request.

[0065] Replay: The process of restoring the recorded data, re-initiating one or N requests, and performing MOCK on a specific downstream node.

[0066] MOCK: During playback, the intercepted sub-call will not be actually called. The Sandbox process intervention capability is used to directly return the return value during recording.

[0067] Elasticsearch: Search server is a distributed, highly scalable, and highly real-time search and data analysis engine. It can easily enable large amounts of data to be searched, analyzed, and explored. Taking full advantage of the horizontal scalability of Elasticsearch can make data more valuable in the production environment. The implementation principle of Elasticsearch is mainly divided into the following steps. First, the user submits the data to the Elasticsearch database, and then the corresponding sentence is segmented through the word segmentation controller, and its weight and word segmentation results are stored in the data. When the user searches for data, the results are ranked and scored according to the weight, and then the returned results are presented to the user.

[0068] It is understandable that the traffic deduplication method provided in the embodiment of the present application can be applied to an electronic device, which can be a terminal or a server; the terminal can specifically be a smart phone, a tablet computer, a computer, a personal digital assistant (PDA), etc.; the server can specifically be an application server or a Web server. It is understandable that a traffic recording platform is running in the electronic device, and the traffic recording platform can be connected to the business server in communication and receive the recorded traffic sent by the business server.

[0069] Figure 1 A flow chart of a traffic deduplication method provided in an embodiment of the present application is shown as follows: Figure 1 As shown, the method includes:

[0070] Step 101: Obtaining traffic data to be deduplicated, the traffic data to be deduplicated includes multiple traffic data; wherein each of the traffic data includes a first identifier, a second identifier, and recorded traffic, the first identifier is generated according to a recording interface corresponding to the recorded traffic, and the second identifier is generated according to a subcall set and a service label corresponding to the recorded traffic;

[0071] Step 102: Deduplication of the to-be-deduplicated traffic data is performed according to the first identifier and the second identifier corresponding to each piece of traffic data.

[0072] In step 101, the traffic data to be deduplicated may be pre-stored in a preset location, for example, in a search server or in a database. The electronic device may obtain traffic data within a preset time period from the search server according to a preset period to form the traffic data to be deduplicated. For example, the preset period may be one day, and the electronic device may obtain all traffic data of the previous day at 10 a.m. every day.

[0073] Since a lot of traffic data is generated in a day, and there is a lot of duplicate traffic data, it is necessary to deduplicate the duplicate traffic data. It can be understood that the so-called duplicate traffic data refers to the traffic data of the same scene corresponding to the same recording interface. Take online shopping as an example: two users submit orders in the same activity on the same online shopping platform. The traffic data corresponding to these two orders are duplicate traffic data.

[0074] The first identifier is an identifier used to characterize the global uniqueness of the recording interface corresponding to the recorded traffic. It can be obtained by calculating the recording interface using the MD5 algorithm, or by calculating the recording interface using a hash algorithm, or by other algorithms. The embodiment of the present application does not make any specific limitations on this.

[0075] The recorded traffic is the service request sent by the user terminal to the service server. The service server generates the service traffic in the process of responding to the service request, and sends the service traffic to the electronic device through a preset request method, which is recorded by the traffic recording platform in the electronic device. It can be understood that one service request can correspond to one recorded traffic. The service server can send the service traffic to the electronic device in the form of an http request. The http request contains a recording interface, so the recording interface can be extracted from the http request.

[0076] The second identifier is used to characterize the global uniqueness of the recorded traffic, which is obtained by calculating the sub-call set and service label corresponding to the recorded traffic according to a preset algorithm. The preset algorithm can be a one-way function algorithm. It is difficult for the one-way function algorithm to have a function output result and infer the specific input data. Among them, the one-way function algorithm can be a cryptographic hash function, specifically an MD5 algorithm, a hash algorithm, etc. Among them, the recorded traffic contains multiple sub-call sets, and the multiple sub-call sets can include main thread sub-calls and asynchronous thread sub-calls.

[0077] The service tag is used to characterize the type of recorded traffic, which can be determined by some specific fields in the recorded traffic. It is understandable that the recorded traffic may correspond to one service tag, or may correspond to multiple service tags, or may not have a corresponding service tag. In the case of no service tag, the electronic device may generate a second identifier according to the sub-call set.

[0078] In step 102, since the same recording interface may include traffic data of multiple scenes, and the same scene may include multiple traffic data, it is necessary to deduplicate the multiple traffic data belonging to the same scene in the same recording interface. Moreover, different recording interfaces are considered to be different traffic even if the traffic data they contain is the same. Therefore, in order to improve the accuracy of traffic deduplication, the first identifier can be used to distinguish the recording interface, and the second identifier can be used to distinguish the traffic data, so that the data of the traffic to be deduplicated can be deduplicated according to the first identifier and the second identifier.

[0079] The embodiment of the present application deduplicates traffic by utilizing a first identifier corresponding to a recording interface and a second identifier generated by a service tag and a sub-call set, thereby improving the efficiency and accuracy of traffic deduplication.

[0080] On the basis of the above embodiment, deduplication of the to-be-deduplicated traffic data according to the first identifier and the second identifier corresponding to each piece of traffic data includes:

[0081] Determine the same traffic data in the recorded traffic according to the first identifier and the second identifier;

[0082] The identical traffic data is partially removed, and only one piece of traffic data is retained, so as to achieve deduplication of the traffic data to be deduplicated.

[0083] In the specific implementation process, the Spark deduplication method can be used for deduplication. Spark includes a variety of deduplication methods, such as distinct deduplication, group by operation, row_number window operation, etc. Distinct deduplication uses one or more fields as the basis for deduplication. When the values ​​corresponding to the fields in two data are the same, the two data are considered to be the same data, and only one of them is retained. The group by operation uses the deduplication column as the aggregation field to achieve the purpose of deduplication through aggregation. For example, the first identifier and the second identifier are used as the aggregation fields, and the traffic data is aggregated according to the first identifier and the second identifier, and only one traffic data after aggregation is retained. In this way, the deduplication operation of the deduplication traffic data is realized.

[0084] In the embodiment of the present application, the first identifier can be used to characterize the uniqueness of the recording interface, and the second identifier can be used to characterize the uniqueness of the sub-call. Therefore, deduplication of the recording traffic is performed using the first identifier and the second identifier, which can improve the accuracy of deduplication.

[0085] Based on the above embodiment, before obtaining the traffic data to be deduplicated, the method further includes:

[0086] Acquire recorded traffic through a jvm sandbox repeater tool, where the recorded traffic includes a subcall set;

[0087] Extracting a recording interface of the recorded traffic and generating a first identifier according to the recording interface;

[0088] Generate a service tag corresponding to the recorded traffic, and generate a second identifier according to the service tag and the sub-call set.

[0089] In the specific implementation process, the recorded traffic is obtained by recording the service request by the jvm sandbox repeater tool running in the electronic device. The relevant explanation of the recorded traffic can be found in the above embodiment and will not be repeated here. When responding to the service request sent by the user end, the service server may make multiple calls. Therefore, the recorded traffic includes traffic data corresponding to multiple sub-calls, and the traffic data corresponding to multiple sub-calls constitute a sub-call set.

[0090] When the service server reports the recorded traffic to the electronic device, the reporting request will carry the recording interface, for example, the recording interface can be extracted by calling the http request through the entry. After obtaining the recording interface, the first identifier corresponding to the recording interface can be generated according to the preset algorithm, and the details can be referred to the above embodiment.

[0091] After obtaining the recorded traffic, the electronic device determines the service tag corresponding to the recorded traffic according to the preset field in the recorded traffic. It is understandable that the correspondence between the field and the service tag can be pre-stored in the electronic device. After obtaining the recorded traffic, each field in the recorded traffic is matched with the pre-set field. If the match is successful, the field in the recorded traffic that is successfully matched is used as the preset field. The service tag of the preset field is determined from the pre-stored correspondence.

[0092] After acquiring the service tag, the electronic device uses a preset algorithm to generate a second identifier for the service tag and the sub-call set. The specific preset algorithm can be found in the above embodiment and will not be described again here.

[0093] The embodiment of the present application deduplicates traffic by utilizing a first identifier corresponding to a recording interface and a second identifier generated by a service tag and a sub-call set, thereby improving the efficiency and accuracy of traffic deduplication.

[0094] Based on the above embodiment, the call type of the subcall can be determined in the following manner:

[0095] The electronic device pre-stores a hash value corresponding to the main thread. The electronic device calculates the hash value of the thread corresponding to the sub-call, and compares the calculated hash value with the stored hash value of the main thread. If they are consistent, the sub-call is determined to be a main thread sub-call. If they are inconsistent, the sub-call is determined to be an asynchronous thread sub-call.

[0096] On the basis of the above embodiment, the generating a second identifier according to the service tag and the sub-call set of the recorded traffic includes:

[0097] Generate a main thread sub-call set according to the sub-call of the main thread sub-call of the call type;

[0098] Generate an asynchronous thread sub-call set according to the sub-call whose call type is an asynchronous thread sub-call;

[0099] Generate a target character string from the main thread sub-call set, the asynchronous thread sub-call set and the service tag according to a preset format;

[0100] The target character string is calculated using a preset algorithm to obtain the second identifier.

[0101] In the specific implementation process, after determining the call type of each sub-call, the electronic device can form a main thread sub-call set with sub-calls belonging to the main thread sub-call, and form an asynchronous thread sub-call set with sub-calls belonging to the asynchronous thread sub-call. Then, the main thread sub-call set, the asynchronous thread sub-call set and the business label are combined to form a large set, namely, the target string. Specifically, the main thread sub-call set, the asynchronous thread sub-call set and the business label can be separated by a comma in English to obtain the target string, namely, the target string = main thread sub-call set, asynchronous thread sub-call set, business label.

[0102] After the target character string is obtained, a preset algorithm such as an MD5 algorithm or a hash algorithm may be used to calculate the target character string to obtain a second identifier.

[0103] The embodiment of the present application can avoid many unnecessary scenarios by dividing the sub-calls into main thread sub-calls and asynchronous thread sub-calls, and combined with business tags, improves the accuracy of deduplication of recorded traffic.

[0104] Based on the above embodiment, after generating the second identifier, the method further includes:

[0105] storing the first identifier, the second identifier, and the recorded traffic as a piece of traffic data in a search server;

[0106] Accordingly, the step of obtaining the traffic data to be deduplicated includes:

[0107] The to-be-deduplicated traffic data within a preset time period is obtained from the search server.

[0108] In the specific implementation process, since a large amount of business traffic can be generated in a business server in a short period of time, the electronic device can obtain a large amount of recorded traffic in real time. If the recorded traffic is deduplicated in real time, the load pressure of the electronic device will increase. In order to reduce the load pressure of the electronic device, after obtaining the first identifier and the second identifier corresponding to the recorded traffic, the embodiment of the present application generates a piece of traffic data for the recorded traffic and the first identifier and the second identifier corresponding to the recorded traffic, and stores the generated traffic data in the search server. It can be understood that the traffic data can also include information such as the timestamp corresponding to the recorded traffic. It can be understood that the timestamp of the recorded traffic can be the time when the recorded traffic is generated, the time when the recorded traffic is stored in the search server, the time when the electronic device receives the recorded traffic, etc.

[0109] The electronic device can obtain the traffic data within the preset time period from the search server, and refer to the obtained traffic data as the traffic data to be deduplicated. The preset time period can be a time period input by the tester, for example, the traffic data from October 1 to 2, 2022; the preset time period can also be a time period determined according to the current time, for example, the traffic data of the day before the date corresponding to the current time.

[0110] It is understandable that in order to reduce the storage pressure of the search server, the traffic data in the search server may be cleaned up regularly. For example, the traffic data two weeks ago may be cleared.

[0111] Since the amount of recorded traffic data is large, in order to reduce the pressure on the recording platform, the preprocessed traffic data is stored in the search server, and then the data of the recorded traffic to be deduplicated is read from the search server to achieve the purpose of reducing the pressure on the recording platform.

[0112] Figure 2 A flow chart of another method for deduplication of traffic provided in an embodiment of the present application is shown in FIG. Figure 2 As shown, the method includes:

[0113] The business server determines whether the sub-call is an asynchronous thread: After the business server generates business traffic according to the business request sent by the user end, it determines whether each sub-call contained in the business traffic is an asynchronous thread, and marks the sub-call. It can be understood that the method for determining whether the thread corresponding to the sub-call is an asynchronous thread can be found in the above embodiment, which will not be repeated here. And "0" can be used to indicate that the thread corresponding to the sub-call is the main thread, and "1" can be used to indicate that the thread corresponding to the sub-call is an asynchronous thread. It can be understood that other marking methods can also be used to mark it, and the embodiments of the present application do not specifically limit this.

[0114] The service server sends service traffic to the traffic recording platform (jvm sandbox repeater) of the electronic device, and specifically sends the service traffic through HTTP request (jvm sandbox repeater module). After receiving the service traffic, the traffic recording platform controller (jvm sandbox repeater console) in the electronic device records the service traffic to obtain the recorded traffic.

[0115] The electronic device obtains the recording interface from the recording traffic sent by the service server, and generates an entry MD5 according to the recording interface.

[0116] The electronic device adds the main thread sub-call set, the asynchronous thread sub-call set and the business label into a large set, connects them with English commas to form a string (ie, the above-mentioned target string), performs MD5 calculation on the string, and obtains a sub-MD5.

[0117] The inlet MD5, sub-MD5 and recorded traffic form a piece of traffic data and store it in elastic search.

[0118] Extract multiple traffic data from elastic search, and use spark to deduplicate the recorded traffic based on the entry MD5 and sub-MD5.

[0119] Store the deduplicated traffic data in a preset database, such as MySQL database.

[0120] The embodiment of the present application controls the traffic by utilizing business tags, and combines the sub-call set of the main thread and the sub-call set of the asynchronous thread to ensure the basic skeleton of the recorded business request, thereby improving the accuracy of deduplication while improving the efficiency of deduplication of recorded traffic.

[0121] Figure 3 The schematic diagram of the structure of a traffic deduplication device provided in the embodiment of the present application is shown in FIG. 1 , and the device may be a module, a program segment or a code on an electronic device. It should be understood that the device is similar to the above-mentioned Figure 1 The method embodiment corresponds to and can be executed Figure 1 The various steps involved in the method embodiment and the specific functions of the device can be found in the above description. To avoid repetition, detailed description is appropriately omitted here. The device includes: a data acquisition module 301 and a deduplication module 302; wherein:

[0122] The data acquisition module 301 is used to acquire the traffic data to be deduplicated, and the traffic data to be deduplicated includes multiple traffic data; wherein each of the traffic data includes a first identifier, a second identifier and recorded traffic, the first identifier is generated according to the recording interface corresponding to the recorded traffic, and the second identifier is generated according to the sub-call set and service label corresponding to the recorded traffic;

[0123] The deduplication module 302 is used to deduplicate the to-be-deduplicated traffic data according to the first identifier and the second identifier corresponding to each piece of traffic data.

[0124] Based on the above embodiment, the deduplication module 302 is specifically used for:

[0125] Determine the same traffic data in the recorded traffic according to the first identifier and the second identifier;

[0126] The identical traffic data is partially removed, and only one piece of traffic data is retained, so as to achieve deduplication of the traffic data to be deduplicated.

[0127] Based on the above embodiment, the device further includes a flow preprocessing module, which is used to:

[0128] Acquire recorded traffic through a jvm sandbox repeater tool, where the recorded traffic includes a subcall set;

[0129] Extracting a recording interface of the recorded traffic and generating a first identifier according to the recording interface;

[0130] Generate a service tag corresponding to the recorded traffic, and generate a second identifier according to the service tag and the sub-call set.

[0131] Based on the above embodiment, the sub-call set includes at least one sub-call; and the call type corresponding to the at least one sub-call is determined in advance by the business server according to the hash value of the thread corresponding to the sub-call; wherein the call type includes a main thread sub-call and an asynchronous thread sub-call.

[0132] Based on the above embodiment, the traffic preprocessing module is specifically used for:

[0133] Generate a main thread sub-call set according to the sub-call of the main thread sub-call of the call type;

[0134] Generate an asynchronous thread sub-call set according to the sub-call whose call type is an asynchronous thread sub-call;

[0135] Generate a target character string from the main thread sub-call set, the asynchronous thread sub-call set and the service tag according to a preset format;

[0136] The target character string is calculated using a preset algorithm to obtain the second identifier.

[0137] Based on the above embodiment, the device further includes a data storage module for:

[0138] storing the first identifier, the second identifier, and the recorded traffic as a piece of traffic data in a search server;

[0139] Accordingly, the traffic preprocessing module is specifically used for:

[0140] The to-be-deduplicated traffic data within a preset time period is obtained from the search server.

[0141] Based on the above embodiment, the traffic preprocessing module is specifically used for:

[0142] Extract preset fields from each flow data;

[0143] A service tag matching the preset field is determined from pre-stored field tag correspondences.

[0144] Figure 4 A schematic diagram of the physical structure of an electronic device provided in an embodiment of the present application, such as Figure 4 As shown, the electronic device includes: a processor (processor) 401, a memory (memory) 402 and a bus 403; wherein,

[0145] The processor 401 and the memory 402 communicate with each other via the bus 403;

[0146] The processor 401 is used to call the program instructions in the memory 402 to execute the methods provided by the above-mentioned method embodiments, for example, including: obtaining the traffic data to be deduplicated, the traffic data to be deduplicated includes multiple traffic data; wherein each of the traffic data includes a first identifier, a second identifier and recorded traffic, the first identifier is generated according to the recording interface corresponding to the recorded traffic, and the second identifier is generated according to the sub-call set and service label corresponding to the recorded traffic; deduplicating the traffic data to be deduplicated according to the first identifier and the second identifier corresponding to each traffic data.

[0147] The processor 401 may be an integrated circuit chip with signal processing capabilities. The processor 401 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. It may implement or execute various methods, steps and logic block diagrams disclosed in the embodiments of the present application. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0148] The memory 402 may include, but is not limited to, random access memory (RAM), read only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable read-only memory (EEPROM), etc.

[0149] The present embodiment discloses a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the methods provided by the above-mentioned method embodiments, for example, including: obtaining traffic data to be deduplicated, and the traffic data to be deduplicated includes multiple traffic data; wherein each of the traffic data includes a first identifier, a second identifier and recorded traffic, the first identifier is generated according to a recording interface corresponding to the recorded traffic, and the second identifier is generated according to a sub-call set and a service label corresponding to the recorded traffic; and deduplicating the traffic data to be deduplicated according to the first identifier and the second identifier corresponding to each traffic data.

[0150] The present embodiment provides a non-transitory computer-readable storage medium, which stores computer instructions, and the computer instructions enable the computer to execute the methods provided by the above-mentioned method embodiments, for example, including: obtaining traffic data to be deduplicated, and the traffic data to be deduplicated includes multiple traffic data; wherein each of the traffic data includes a first identifier, a second identifier and recorded traffic, the first identifier is generated according to a recording interface corresponding to the recorded traffic, and the second identifier is generated according to a sub-call set and a service label corresponding to the recorded traffic; deduplicating the traffic data to be deduplicated according to the first identifier and the second identifier corresponding to each traffic data.

[0151] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0152] In addition, the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0153] Furthermore, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0154] In this document, relational terms such as first and second, etc. are used merely to distinguish one entity or operation from another entity or operation, but do not necessarily require or imply any such actual relationship or order between these entities or operations.

[0155] The above description is only an embodiment of the present application and is not intended to limit the scope of protection of the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the scope of protection of the present application.

Claims

1. A traffic deduplication method, characterized in that: include: Obtain traffic data to be deduplicated, the traffic data to be deduplicated includes multiple traffic data; wherein each of the traffic data includes a first identifier, a second identifier and recorded traffic, the first identifier is generated according to a recording interface corresponding to the recorded traffic, and the second identifier is generated by calculation according to a preset algorithm based on a sub-call set and a service label corresponding to the recorded traffic; the first identifier is an identifier used to characterize the global uniqueness of the recording interface corresponding to the recorded traffic; the second identifier is an identifier used to characterize the global uniqueness of the recorded traffic; the service label is used to characterize the type of recorded traffic; The to-be-deduplicated traffic data is deduplicated according to the first identifier and the second identifier corresponding to each piece of traffic data.

2. The method according to claim 1, characterized in that Deduplication of the to-be-deduplicated traffic data according to the first identifier and the second identifier corresponding to each piece of traffic data includes: Determine the same traffic data in the recorded traffic according to the first identifier and the second identifier; The identical traffic data is partially removed, and only one piece of traffic data is retained, so as to achieve deduplication of the traffic data to be deduplicated.

3. The method according to claim 1, characterized in that Before obtaining the traffic data to be deduplicated, the method further includes: Acquire recorded traffic through a jvm sandbox repeater tool, where the recorded traffic includes a subcall set; Extracting a recording interface of the recorded traffic and generating a first identifier according to the recording interface; Generate a service tag corresponding to the recorded traffic, and generate a second identifier according to the service tag and the sub-call set.

4. The method according to claim 3, characterized in that The sub-call set includes at least one sub-call; and the call type corresponding to the at least one sub-call is determined in advance by the business server according to the hash value of the thread corresponding to the sub-call; wherein the call type includes a main thread sub-call and an asynchronous thread sub-call.

5. The method according to claim 4, characterized in that The generating a second identifier according to the service tag and the sub-call set includes: Generate a main thread sub-call set according to the sub-call of the main thread sub-call of the call type; Generate an asynchronous thread sub-call set according to the sub-call whose call type is an asynchronous thread sub-call; Generate a target character string from the main thread sub-call set, the asynchronous thread sub-call set and the service tag according to a preset format; The target character string is calculated using a preset algorithm to obtain the second identifier.

6. The method according to claim 3, characterized in that: The generating a service label corresponding to the recorded traffic includes: Extract preset fields from each flow data; A service tag matching the preset field is determined from pre-stored field tag correspondences.

7. The method according to any one of claims 3 to 6, characterized in that: After generating the second identifier, the method further includes: storing the first identifier, the second identifier, and the recorded traffic as a piece of traffic data in a search server; Accordingly, the step of obtaining the traffic data to be deduplicated includes: The to-be-deduplicated traffic data within a preset time period is obtained from the search server.

8. A flow deduplication device, characterized in that: include: A data acquisition module, used to acquire traffic data to be deduplicated, wherein the traffic data to be deduplicated includes multiple traffic data; wherein each traffic data includes a first identifier, a second identifier and recorded traffic, wherein the first identifier is generated according to a recording interface corresponding to the recorded traffic, and the second identifier is generated by calculation according to a preset algorithm based on a sub-call set and a service tag corresponding to the recorded traffic; the first identifier is an identifier used to characterize the global uniqueness of the recording interface corresponding to the recorded traffic; the second identifier is an identifier used to characterize the global uniqueness of the recorded traffic; the service tag is used to characterize the type of recorded traffic; The deduplication module is used to deduplicate the traffic data to be deduplicated according to the first identifier and the second identifier corresponding to each piece of traffic data.

9. An electronic device, characterized in that: include: processor, memory and bus, wherein, The processor and the memory communicate with each other via the bus; The memory stores program instructions executable by the processor, and the processor can execute the method according to any one of claims 1 to 7 by calling the program instructions.

10. A non-transitory computer-readable storage medium, characterized in that: The non-transitory computer-readable storage medium stores computer instructions, which, when executed by a computer, cause the computer to perform the method according to any one of claims 1 to 7.

Images