Method, device, equipment and storage medium for deduplication of massive requests

Abstract

The present application relates to the field of computer technology, and discloses a method, device, equipment and storage medium for deduplication of massive requests to solve the problem of low deduplication accuracy. The method comprises: obtaining a link message of each initial user request in an initial user request set, each link message being generated based on a link node providing services for the same initial user request; then converting and processing each link message respectively to obtain a corresponding link tag, and based on each obtained link tag, eliminating duplicate user requests in the initial user request set to obtain a target user request set. The link tag of each link message integrates the jump information between different link nodes and the actual description information of each link node, and more truly reflects the actual business logic of the user request. Deduplication of massive user requests is performed based on each link tag, thereby overcoming the technical defects of the deduplication model and improving the deduplication accuracy and versatility.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method, device, equipment and storage medium for deduplicating massive requests. Background Art

[0002] With the increasing growth of Internet business, various platforms have begun to build microservice architectures to support the huge business volume. As the business volume grows, removing duplicate user requests has become a key technical means to reduce platform maintenance costs.

[0003] In the related art, a deduplication model is usually used to deduplicate massive user requests. However, the deduplication model tends to ignore the correlation between surface information (such as the parameters of user requests, the response results of user requests) and the actual business meaning, resulting in the filtering out of user requests that contain the same surface information but different actual business meanings; secondly, if the inclusion of the same request parameters is used as the criterion for repeated user requests, the deduplication model will retain user requests with different parameter combinations but the same request response results, which will reduce the deduplication accuracy. Moreover, as the meaning and quantity of surface information change, the model architecture and model parameters of the deduplication model need to be adaptively adjusted, further increasing the platform maintenance cost.

[0004] In view of this, it is necessary to design a new deduplication method, device, equipment and storage medium for massive requests to overcome the above-mentioned defects. Summary of the invention

[0005] The embodiments of the present application provide a method, apparatus, device and storage medium for deduplication of massive requests, so as to solve the problem of low deduplication accuracy.

[0006] The specific technical solutions provided by the embodiments of this application are as follows:

[0007] In a first aspect, an embodiment of the present application provides a method for deduplicating massive requests, including:

[0008] Obtaining a link message of each initial user request in the initial user request set, where each link message is generated based on a link node that provides services for the same initial user request;

[0009] Convert and process each link message respectively to obtain the corresponding link tag;

[0010] Based on the obtained link tags, repeated user requests in the initial user request set are eliminated to obtain a target user request set.

[0011] Optionally, a link message includes at least one association information group;

[0012] Wherein, an associated information group is generated by performing the following operations:

[0013] In response to monitoring an operation that any link node provides a service for an initial user request, obtaining at least a request identifier of the initial user request, a node identifier of the any link node, and a probe identifier of a probe preset on the any link node;

[0014] The request identifier of the initial user request, the node identifier of the arbitrary link node, and the probe identifier of the probe preset on the arbitrary link node are associated and stored to generate a corresponding association information group.

[0015] Optionally, the link messages are converted respectively to obtain corresponding link tags, wherein a link tag is obtained by performing the following operations:

[0016] Aggregate at least one associated information group included in the link message to obtain at least one aggregated sub-information;

[0017] Performing splicing processing on the at least one aggregated sub-information to obtain target aggregated information;

[0018] Mapping processing is performed on the target aggregation information to obtain a corresponding link label.

[0019] Optionally, the aggregating at least one associated information group included in the link message to obtain at least one aggregated sub-information includes:

[0020] Aggregating the associated information groups containing the same node identifier in the one link message to obtain the at least one aggregated sub-information;

[0021] Each aggregated sub-information includes a node identifier and at least one probe identifier.

[0022] Optionally, if one aggregate sub-information contains multiple probe identifiers, the probe identifiers in the one aggregate sub-information are sorted according to the order of the size of the probe identifiers.

[0023] Optionally, the removing repeated user requests from the initial user request set based on the obtained link tags to obtain the target user request set includes:

[0024] In a loop, the following operations are performed for the candidate user request set of the current round until there are no duplicate candidate user requests in the candidate user request set of the next round, thereby obtaining the target user request set:

[0025] Arrange and combine the candidate user request set of the current round to obtain a plurality of user request pairs, each user request pair including two candidate user requests;

[0026] The following operations are performed for each user request pair: obtaining link tags of two candidate user requests in a user request pair; if the link tag similarity between the two link tags exceeds a set threshold, determining that the two candidate user requests are duplicate user requests, and eliminating any candidate user request;

[0027] The candidate user request set after deduplication in the current round is used as the candidate user request set for the next round.

[0028] Optionally, determine the candidate user request set for the current round by performing the following operations:

[0029] If it is the first cycle, the initial user request set is used as the candidate user request set for the current round;

[0030] If it is not the first cycle, the candidate user request set after deduplication in the previous round is used as the candidate user request set in the current round.

[0031] Optionally, the removing repeated user requests from the initial user request set based on the obtained link tags to obtain the target user request set includes:

[0032] In a cyclic manner, the following operations are performed for different types of candidate user request sets respectively until there are no duplicate candidate user requests in the candidate user request sets of each type in the next round, and the corresponding target user request sets are obtained:

[0033] A set of candidate user requests in the current round is arranged and combined to obtain multiple user request pairs, each of which contains two candidate user requests;

[0034] The following operations are performed for each user request pair: obtaining link tags of two candidate user requests in a user request pair; if the link tag similarity between the two link tags exceeds a set threshold, determining that the two candidate user requests are duplicate user requests, and eliminating any candidate user request;

[0035] A candidate user request set after deduplication in the current round is used as a candidate user request set for the next round.

[0036] Optionally, determine a candidate user request set for the current round by performing the following operations:

[0037] If it is the first cycle, an initial user request set of a type is used as a candidate user request set for the current round;

[0038] If it is not the first cycle, a candidate user request set after deduplication in the previous round is used as a candidate user request set in the current round.

[0039] Optionally, the link mark similarity between the two link marks is obtained by performing the following operations:

[0040] Using a character deduplication algorithm, compare characters at the same position on the two link tags one by one to obtain the similarity of the link tags; or,

[0041] A hash algorithm is used to obtain hash values ​​of the two link labels respectively, and Hamming distance processing is performed on the two hash values ​​to obtain the link label similarity.

[0042] In a second aspect, the embodiment of the present application further provides a device for deduplicating massive requests, including:

[0043] An acquisition unit, used to acquire a link message of each initial user request in the initial user request set, each link message being generated based on a link node providing services for the same initial user request;

[0044] A deduplication unit, used to convert and process each link message respectively to obtain a corresponding link tag;

[0045] Based on the obtained link tags, repeated user requests in the initial user request set are eliminated to obtain a target user request set.

[0046] Optionally, a link message includes at least one associated information group, wherein the apparatus further includes a generating unit, and the generating unit generates an associated information group by performing the following operations:

[0047] In response to monitoring an operation that any link node provides a service for an initial user request, obtaining at least a request identifier of the initial user request, a node identifier of the any link node, and a probe identifier of a probe preset on the any link node;

[0048] The request identifier of the initial user request, the node identifier of the arbitrary link node, and the probe identifier of the probe preset on the arbitrary link node are associated and stored to generate a corresponding association information group.

[0049] Optionally, the deduplication unit obtains a link tag by performing the following operations:

[0050] Aggregate at least one associated information group included in the link message to obtain at least one aggregated sub-information;

[0051] Performing splicing processing on the at least one aggregated sub-information to obtain target aggregated information;

[0052] Mapping processing is performed on the target aggregation information to obtain a corresponding link label.

[0053] Optionally, the deduplication unit is used to:

[0054] Aggregating the associated information groups containing the same node identifier in the one link message to obtain the at least one aggregated sub-information;

[0055] Each aggregated sub-information includes a node identifier and at least one probe identifier.

[0056] Optionally, if one aggregate sub-information contains multiple probe identifiers, the probe identifiers in the one aggregate sub-information are sorted according to the order of the size of the probe identifiers.

[0057] Optionally, the deduplication unit is used to:

[0058] In a loop, the following operations are performed for the candidate user request set of the current round until there are no duplicate candidate user requests in the candidate user request set of the next round, thereby obtaining the target user request set:

[0059] Arrange and combine the candidate user request set of the current round to obtain a plurality of user request pairs, each user request pair including two candidate user requests;

[0060] The following operations are performed for each user request pair: obtaining link tags of two candidate user requests in a user request pair, and if the link tag similarity between the two link tags exceeds a set threshold, determining that the two candidate user requests are duplicate user requests, and eliminating any candidate user request;

[0061] The candidate user request set after deduplication in the current round is used as the candidate user request set for the next round.

[0062] Optionally, the deduplication unit determines the candidate user request set of the current round by performing the following operations:

[0063] If it is the first cycle, the initial user request set is used as the candidate user request set for the current round;

[0064] If it is not the first cycle, the candidate user request set after deduplication in the previous round is used as the candidate user request set in the current round.

[0065] Optionally, the deduplication unit is used to:

[0066] In a cyclic manner, the following operations are performed for different types of candidate user request sets respectively until there are no duplicate candidate user requests in the candidate user request sets of each type in the next round, and the corresponding target user request sets are obtained:

[0067] A set of candidate user requests in the current round is arranged and combined to obtain multiple user request pairs, each of which contains two candidate user requests;

[0068] The following operations are performed for each user request pair: obtaining link tags of two candidate user requests in a user request pair; if the link tag similarity between the two link tags exceeds a set threshold, determining that the two candidate user requests are duplicate user requests, and eliminating any candidate user request;

[0069] A candidate user request set after deduplication in the current round is used as a candidate user request set for the next round.

[0070] Optionally, the deduplication unit determines a candidate user request set for the current round by performing the following operations:

[0071] If it is the first cycle, an initial user request set of a type is used as a candidate user request set for the current round;

[0072] If it is not the first cycle, a candidate user request set after deduplication in the previous round is used as a candidate user request set in the current round.

[0073] Optionally, the deduplication unit obtains the link label similarity between the two link labels by performing the following operations:

[0074] Using a character deduplication algorithm, compare characters at the same position on the two link tags one by one to obtain the similarity of the link tags; or,

[0075] A hash algorithm is used to obtain hash values ​​of the two link labels respectively, and Hamming distance processing is performed on the two hash values ​​to obtain the link label similarity.

[0076] In a third aspect, an embodiment of the present application further provides a computer device, comprising a processor and a memory, wherein the memory stores program code, and when the program code is executed by the processor, the processor executes the steps of any one of the above-mentioned methods for deduplication of massive requests.

[0077] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, comprising a program code. When the program code is executed on a computer device, the program code is used to enable the computer device to execute the steps of any one of the above-mentioned methods for deduplicating massive requests.

[0078] The beneficial effects of this application are as follows:

[0079] In the embodiment of the present application, the link message of each initial user request in the initial user request set is obtained, and each link message is generated based on the link node that provides services for the same initial user request; then each link message is converted and processed respectively to obtain the corresponding link tag, and based on each link tag obtained, the repeated user requests in the initial user request set are eliminated to obtain the target user request set. In the deduplication method for massive requests provided in the embodiment of the present application, each link message is generated based on the link node that provides services for the same initial user request. Therefore, each link message reflects the link node tracking information in the process of providing services for the corresponding initial user request, and the link tag of each link message integrates the jump information between different link nodes and the actual description information of each link node, which more truly reflects the actual business logic of the user request. Using the obtained link tags as the criterion for repeated user requests, the technical defects of the deduplication model are overcome and the purpose of improving the accuracy and versatility of deduplication is achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0080] Figure 1a A flowchart of a method for deduplicating massive requests provided in an embodiment of the present application;

[0081] Figure 1b A logical schematic diagram of generating a link message provided in an embodiment of the present application;

[0082] Figure 1c A schematic diagram of a process for generating a link tag provided in an embodiment of the present application;

[0083] Figure 2 A schematic diagram of the process of the first deduplication method provided in the embodiment of the present application;

[0084] Figure 3 A schematic diagram of a second deduplication method provided in an embodiment of the present application;

[0085] Figure 4 A schematic diagram of the structure of a device for recognizing keywords provided in an embodiment of the present application;

[0086] Figure 5 A schematic diagram of the structure of a computer device provided in an embodiment of the present application;

[0087] Figure 6 A schematic diagram of the hardware composition structure of a computing device provided for applying an embodiment of the present application. DETAILED DESCRIPTION

[0088] In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the technical scheme of the present application, rather than all of the embodiments. Based on the embodiments recorded in the application documents, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the technical scheme of the present application.

[0089] With the growing Internet business, various platforms have begun to build microservice architectures to support the huge business volume. As the business volume grows, removing duplicate user requests has become a key technical means to reduce platform maintenance costs.

[0090] In related technologies, a deduplication model is usually used to deduplicate massive user requests.

[0091] For example, using the inclusion of the same parameters as the criterion for determining repeated user requests, a group of user requests shown in Table 1 are deduplicated to obtain a target user request set as shown in Table 2.

[0092] A group of user requests shown in Table 1 includes a parameter a of a numeric type. For the numeric type parameter, a boundary analysis method is used to set multiple boundary conditions such as a minimum value, a maximum value, a null value, a positive value and a negative value (select a positive number from multiple positive numbers that are not the maximum value as the boundary value at one end, and select a negative number from multiple negative numbers that are not the minimum value as the boundary value at the other end). User requests whose parameters meet various boundary conditions are determined to be different types of user requests, and user requests whose parameters do not meet any boundary condition are determined to be repeated user requests.

[0093] Table 1

[0094]

[0095] Table 2

[0096]

[0097] For another example, taking the inclusion of at least one identical field as the criterion for determining duplicate user requests, a group of user requests shown in Table 3 are deduplicated to obtain a target user request set as shown in Table 4.

[0098] A set of user requests shown in Table 3 includes response results in the JavaScript Object Notation (JSON) string format, each of which includes at least a code field, a message (msg) field, and a data field, and some response results also include a count field. The code field indicates the response code, the msg field indicates the error message, the data field indicates the response data, and the count field indicates the numeric response data.

[0099] Table 3

[0100] User request ID Response results User request 0 {"code":301,"msg":"redirect","data":{}} User Request 1 {"code":200,"msg":"","data":{} User Request 2 {"code":500,"msg":"inner error","data":{}} User Request 3 {"code":200,"msg":"","data":{"count":0}} User Request 4 {"code":500,"msg":"inner exception","data":{}} User Request 5 {"code":200,"msg":"","data":{"count":1}} User Request 6 {"code":200,"msg":"","data":{"count":5}} User Request 7 {"code":200,"msg":"","data":{"count":100}}

[0101] Table 4

[0102]

[0103]

[0104] However, the deduplication model tends to ignore the relationship between surface information and actual business meaning, resulting in the filtering out of user requests that contain the same surface information but different actual business meanings; secondly, if the inclusion of the same request parameters is used as the criterion for repeated user requests, the deduplication model will retain user requests with different parameter combinations but the same request response results, which reduces the deduplication accuracy. Moreover, as the meaning and quantity of surface information change, the model architecture and model parameters of the deduplication model need to be adaptively adjusted, further increasing the platform maintenance cost.

[0105] In order to solve the problem of low deduplication accuracy, the embodiment of the present application proposes a method for deduplication of massive requests. The method can be applied to various application scenarios such as automated testing scenarios and reverse automated testing scenarios, and deduplication is performed on massive user requests in each application scenario to ensure the normal operation of the test work.

[0106] See also Figure 1a The process diagram shown in FIG. 1 includes:

[0107] S101: Acquire a link message of each initial user request in an initial user request set, where each link message is generated based on a link node that provides services for the same initial user request.

[0108] In the microservice architecture, the process of providing services for an initial user request is called a request chain. The request chain includes multiple distributed services called in response to the initial user request and multiple code algorithms called when executing each distributed service.

[0109] In order to achieve the purpose of tracking the request link, the unique request identifier of the initial user request is transparently transmitted between the called distributed services and code algorithms to generate the corresponding link message. Among them, a link message contains at least one associated information group. The following operations are performed to generate an associated information group in the link message:

[0110] In response to monitoring an operation that any link node provides a service for an initial user request, at least obtaining a request identifier of the initial user request, a node identifier of the link node, and a probe identifier of a probe preset on the link node;

[0111] The request identifier of the initial user request, the node identifier of the link node, and the probe identifier of the probe preset on the link node are then associated and stored to generate a corresponding association information group.

[0112] When providing services for an initial user request, if multiple link nodes are called, the link message will contain multiple associated information groups, and each associated information group stores the same request identifier. The link node refers to the code algorithm called to provide services for the initial user request. In order to monitor the execution of special statements such as entry statements, exit statements, and judgment statements in the code algorithm, corresponding probes are also preset on each special statement for monitoring.

[0113] Table 5 shows some special statements. When executing the if / else judgment statement, if the code segment of if is executed, probe 1 is triggered; if the code segment of else is executed, probe 2 is triggered.

[0114] Table 5

[0115] Statement types for special statements Description of special statements Entry Statement Entry of code algorithm Decision Statement If / else, switch, goto and other judgment statements Exit Statement Exit statements such as return and throw exception

[0116] like Figure 1b In the logical schematic diagram shown, in the process of providing service for the initial user request 1, link nodes A~C are called in sequence to generate link message 1 {[1,A,1],[1,A,2],[1,B,2],[1,C,2]}; in the process of providing service for the initial user request 2, link nodes A~C are called in sequence to generate link message 2 {[2,A,4],[2,B,3],[2,B,4],[2,C,2],[2,C,3]}.

[0117] Assuming that the number of initial user requests is N, and the number of link nodes called by each request is M, the data volume of the link messages of all initial user requests is M*N, and the data volume of the link messages far exceeds the number of initial user requests, especially in the scenario where the request link is more complex, the number of initial user requests will be greater. Therefore, in order to prevent the impact on the provision of services for each initial user request, the embodiment of the present application adopts asynchronous methods such as asynchronous logs and asynchronous message queues to output the corresponding link messages.

[0118] S102: Convert each link message respectively to obtain a corresponding link tag.

[0119] like Figure 1c As shown, taking a link message as an example, the following operations are performed on the link message to obtain the corresponding link tag:

[0120] S1021: Aggregate at least one associated information group included in the link message to obtain at least one aggregated sub-information.

[0121] The associated information groups containing the same node identifier in the link message are aggregated to obtain at least one aggregated sub-information; wherein each aggregated sub-information includes a node identifier and at least one probe identifier.

[0122] If an aggregate sub-information contains multiple probe identifiers, the probe identifiers in the aggregate sub-information are sorted according to the order of the size of the probe identifiers.

[0123] S1022: Perform splicing processing on at least one aggregated sub-information to obtain target aggregated information.

[0124] S1023: Map the target aggregation information to obtain a corresponding link label.

[0125] For example, link message 3 {[3,A,1],[3,A,2],[3,B,4],[3,B,2],[3,C,3]} is aggregated to obtain three aggregate sub-information ([A,1],[A,2]), ([B,2],[B,4]) and ([C,3]); each aggregate sub-information is concatenated to obtain target aggregate information {(A-1,2)(B-2,4)(C-3)}, and the message digest algorithm (Message-Digest Algorithm 5, MD5) is executed on the target aggregate information to obtain the link tag fbd2dc72528299e7e9a107312a86ee75.

[0126] S103: Based on the obtained link tags, duplicate user requests in the initial user request set are eliminated to obtain a target user request set.

[0127] Optionally, the present application embodiment provides the following two deduplication methods:

[0128] Deduplication method 1: Deduplication is performed on an initial set of user requests.

[0129] See also Figure 2 The process diagram shown in the figure shows that the execution process of the first deduplication method is as follows:

[0130] S201: Arrange and combine the candidate user request set of the current round to obtain multiple user request pairs, each user request pair including two candidate user requests;

[0131] S202: performing the following operations for each user request pair: obtaining link tags of two candidate user requests in a user request pair, and if the link tag similarity between the two link tags exceeds a set threshold, determining that the two candidate user requests are duplicate user requests, and eliminating any candidate user request;

[0132] S203: Using the candidate user request set after deduplication in the current round as the candidate user request set for the next round;

[0133] S204: Determine whether there are repeated user requests in the candidate user request set for the next round. If so, return to step 201; otherwise, output the candidate user request set for the next round as the target user request set.

[0134] Among them, when executing step 201, if it is the first cycle, the initial user request set is used as the candidate user request set of the current round; if it is not the first cycle, the candidate user request set after deduplication in the previous round is used as the candidate user request set of the current round.

[0135] Deduplication method 2: Deduplication is performed on different types of initial user request sets.

[0136] For example, in the grouping scenario, the second deduplication method is used to analyze user requests from different regions separately. Figure 3 The process diagram shown in the figure shows that the execution process of the second deduplication method is as follows:

[0137] S301: Arrange and combine a set of candidate user requests in the current round to obtain multiple user request pairs, each of which contains two candidate user requests;

[0138] S302: performing the following operations for each user request pair: obtaining link tags of two candidate user requests in a user request pair, and if the link tag similarity between the two link tags exceeds a set threshold, determining that the two candidate user requests are duplicate user requests, and eliminating any candidate user request;

[0139] S303: Using the candidate user request set after deduplication in the current round as the candidate user request set for the next round;

[0140] S304: Determine whether a candidate user request set of each type for the next round is obtained. If so, execute step 305; otherwise, return to step 301;

[0141] S305: Determine whether there are no repeated user requests in the candidate user request sets of each type in the next round. If so, output the candidate user request sets of each type in the next round as the corresponding target user request sets; otherwise, return to step 301.

[0142] Among them, when executing step 301, if it is the first round of loop, an initial user request set of one type is used as a candidate user request set for the current round; if it is not the first round of loop, a candidate user request set after deduplication in the previous round is used as a candidate user request set for the current round.

[0143] Regardless of which of the above deduplication methods is used, the link tag similarity between two link tags can be obtained by performing the following operations:

[0144] Operation 1: Use a character deduplication algorithm to compare the characters at the same position on the two link tags one by one to obtain the link tag similarity.

[0145] For example, a character deduplication algorithm is used to compare the three link labels shown in Table 6, where link labels 1 and 3 are completely consistent. Therefore, the initial user request 1 or the initial user request 3 is removed, and the target user request set after deduplication is shown in Table 7.

[0146] Table 6

[0147] Initial user request ID Link Marking Initial User Request 1 fbd2dc72528299e7e9a107312a86ee75 Initial User Request 2 692a16cdc9b343d4700bae3aae2364d6 Initial User Request 3 fbd2dc72528299e7e9a107312a86ee75

[0148] Table 7

[0149] Initial user request ID Link Marking Initial User Request 1 fbd2dc72528299e7e9a107312a86ee75 Initial User Request 2 692a16cdc9b343d4700bae3aae2364d6

[0150] Alternatively, operation 2: using a hash algorithm to obtain hash values ​​of two link labels respectively, and performing Hamming distance processing on the two hash values ​​to obtain link label similarity.

[0151] For example, a hash algorithm is used to perform hash calculations on the three link labels shown in Table 6 to obtain corresponding hash values, and then the Hamming distance between any two hash values ​​is calculated. If the Hamming distance between link labels 1 and 3 is less than 2, it is determined that the initial user request 1 and the initial user request 3 are duplicate user requests. Therefore, the initial user request 1 or the initial user request 3 is removed, and the target user request set after deduplication is shown in Table 7.

[0152] In the deduplication method for massive requests provided in the embodiment of the present application, each link message is generated based on the link node that provides services for the same initial user request. Therefore, each link message reflects the link node tracking information in the process of providing services for the corresponding initial user request, and the link tag of each link message integrates the jump information between different link nodes and the actual description information of each link node, which more truly reflects the actual business logic of the user request. The obtained link tags are used as the judgment criteria for repeated user requests, thereby overcoming the technical defects of the deduplication model and improving the accuracy and versatility of deduplication.

[0153] Based on the same inventive concept as the above method embodiment, in the embodiment of the present invention, a device for deduplicating massive requests is provided, see Figure 4 As shown, the device 400 at least includes an acquisition unit 401, a deduplication unit 402 and a generation unit 403, wherein:

[0154] An acquisition unit 401 is used to acquire a link message of each initial user request in the initial user request set, where each link message is generated based on a link node that provides services for the same initial user request;

[0155] A deduplication unit 402, used to convert each link message to obtain a corresponding link tag;

[0156] Based on the obtained link tags, repeated user requests in the initial user request set are eliminated to obtain a target user request set.

[0157] Optionally, a link message includes at least one associated information group, wherein the apparatus 400 further includes a generating unit 403, and the generating unit generates an associated information group by performing the following operations:

[0158] In response to monitoring an operation that any link node provides a service for an initial user request, obtaining at least a request identifier of the initial user request, a node identifier of the any link node, and a probe identifier of a probe preset on the any link node;

[0159] The request identifier of the initial user request, the node identifier of the arbitrary link node, and the probe identifier of the probe preset on the arbitrary link node are associated and stored to generate a corresponding association information group.

[0160] Optionally, the deduplication unit 402 obtains a link tag by performing the following operations:

[0161] Aggregate at least one associated information group included in the link message to obtain at least one aggregated sub-information;

[0162] Performing splicing processing on the at least one aggregated sub-information to obtain target aggregated information;

[0163] Mapping processing is performed on the target aggregation information to obtain a corresponding link label.

[0164] Optionally, the deduplication unit 402 is used to:

[0165] Aggregating the associated information groups containing the same node identifier in the one link message to obtain the at least one aggregated sub-information;

[0166] Each aggregated sub-information includes a node identifier and at least one probe identifier.

[0167] Optionally, if one aggregate sub-information contains multiple probe identifiers, the probe identifiers in the one aggregate sub-information are sorted according to the order of the size of the probe identifiers.

[0168] Optionally, the deduplication unit 402 is used to:

[0169] In a loop, the following operations are performed for the candidate user request set of the current round until there are no duplicate candidate user requests in the candidate user request set of the next round, thereby obtaining the target user request set:

[0170] Arrange and combine the candidate user request set of the current round to obtain a plurality of user request pairs, each user request pair including two candidate user requests;

[0171] The following operations are performed for each user request pair: obtaining link tags of two candidate user requests in a user request pair; if the link tag similarity between the two link tags exceeds a set threshold, determining that the two candidate user requests are duplicate user requests, and eliminating any candidate user request;

[0172] The candidate user request set after deduplication in the current round is used as the candidate user request set for the next round.

[0173] Optionally, the deduplication unit 402 determines the candidate user request set of the current round by performing the following operations:

[0174] If it is the first cycle, the initial user request set is used as the candidate user request set for the current round;

[0175] If it is not the first cycle, the candidate user request set after deduplication in the previous round is used as the candidate user request set in the current round.

[0176] Optionally, the deduplication unit 402 is used to:

[0177] In a cyclic manner, the following operations are performed for different types of candidate user request sets respectively until there are no duplicate candidate user requests in the candidate user request sets of each type in the next round, and the corresponding target user request sets are obtained:

[0178] A set of candidate user requests in the current round is arranged and combined to obtain multiple user request pairs, each of which contains two candidate user requests;

[0179] The following operations are performed for each user request pair: obtaining link tags of two candidate user requests in a user request pair; if the link tag similarity between the two link tags exceeds a set threshold, determining that the two candidate user requests are duplicate user requests, and eliminating any candidate user request;

[0180] A candidate user request set after deduplication in the current round is used as a candidate user request set for the next round.

[0181] Optionally, the deduplication unit 402 determines a candidate user request set in the current round by performing the following operations:

[0182] If it is the first cycle, an initial user request set of a type is used as a candidate user request set for the current round;

[0183] If it is not the first cycle, a candidate user request set after deduplication in the previous round is used as a candidate user request set in the current round.

[0184] Optionally, the deduplication unit 402 obtains the link label similarity between the two link labels by performing the following operations:

[0185] Using a character deduplication algorithm, compare characters at the same position on the two link tags one by one to obtain the similarity of the link tags; or,

[0186] A hash algorithm is used to obtain hash values ​​of the two link labels respectively, and Hamming distance processing is performed on the two hash values ​​to obtain the link label similarity.

[0187] Based on the same inventive concept as the above method embodiment, the embodiment of the present invention further provides a computer device, referring to Figure 5 As shown in the structural diagram, the computer device may include at least one processor 501 and at least one memory 502. The memory 502 stores program codes. When the program codes are executed by the processor 501, the processor 501 executes the steps in the method for deduplicating massive requests described above in this specification. For example, the processor 501 may execute the following steps: Figure 1a Follow the steps shown in .

[0188] Refer to the following Figure 6 hereinafter, a computing device 600 according to this embodiment of the present application is described. Figure 6 The computing device 600 is merely an example and should not limit the functionality and scope of use of the embodiments of the present application.

[0189] like Figure 6 As shown, the computing device 600 is in the form of a general computing device. The components of the computing device 600 may include but are not limited to: at least one processing unit 601, at least one storage unit 602, and a bus 603 connecting different system components (including the storage unit 602 and the processing unit 601).

[0190] Bus 603 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a processor, or a local bus using any of a variety of bus architectures.

[0191] The storage unit 602 may include a readable medium in the form of a volatile memory, such as a random access memory (RAM) 6021 and / or a cache memory unit 6022 , and may further include a read-only memory (ROM) 6023 .

[0192] The storage unit 602 may also include a program / utility 6025 having a set (at least one) of program modules 6024, such program modules 6024 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0193] The computing device 600 may also communicate with one or more external devices 6904 (e.g., keyboards, pointing devices, etc.), one or more devices that enable a user to interact with the computing device 600, and / or any device that enables the computing device 600 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via input / output (I / O) interface 605. Furthermore, the computing device 600 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via network adapter 606. Figure 6 As shown, the network adapter 606 communicates with other modules for the computing device 600 via the bus 603. It should be understood that, although not shown in the figure, other hardware and / or software modules can be used in conjunction with the computing device 600, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0194] Based on the same inventive concept as the above method embodiment, various aspects of the method for deduplicating a large number of requests provided in the present application can also be implemented in the form of a program product, which includes a program code. When the program code is run on a computer device, the program code is used to enable the computer device to execute the steps of the method for deduplicating a large number of requests described above in this specification. For example, the computer device can execute the following steps: Figure 1a Follow the steps shown in .

[0195] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0196] The program product for business control of the embodiment of the present application can adopt a portable compact disk read-only memory (CD-ROM) and include program code, and can be run on a computing device. However, the program product of the present application is not limited to this. In this document, a readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with a command execution system, device or device.

[0197] The readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, wherein the readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The readable signal medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with a command execution system, apparatus, or device.

[0198] The program code contained on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

[0199] Program code for performing the operations of the present application may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user equipment, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device via any type of network including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0200] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0201] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A method for deduplicating massive requests, characterized in that: include: Obtaining a link message of each initial user request in the initial user request set, where each link message is generated based on a link node that provides services for the same initial user request; Convert and process each link message respectively to obtain the corresponding link tag; In a cyclic manner, multiple rounds of deduplication are performed on an initial user request set to obtain a target user request set from which all duplicate requests have been removed, wherein multiple user request pairs are obtained by permutation and combination of the candidate user request set of the current round, and the following operations are performed respectively: when the link tag similarity between two candidate user requests in the user request pair exceeds a set threshold, any candidate user request is removed, and the candidate user request set of the current round is the initial user request set or the candidate user request set after deduplication in the previous round; Alternatively, a cyclic manner is adopted to perform multiple rounds of deduplication on candidate user request sets of different types, and obtain multiple target user request sets from which all duplicate requests have been eliminated, wherein multiple user request pairs are obtained by permuting and combining candidate user request sets of one type in the current round, and the following operations are performed respectively: when the link label similarity between two candidate user requests in the user request pair exceeds a set threshold, any candidate user request is eliminated, and the candidate user request set of one type in the current round is an initial user request set of one type or a candidate user request set of one type after deduplication in the previous round.

2. The method according to claim 1, characterized in that A link message includes at least one associated information group; Wherein, an associated information group is generated by performing the following operations: In response to monitoring an operation that any link node provides a service for an initial user request, obtaining at least a request identifier of the initial user request, a node identifier of the any link node, and a probe identifier of a probe preset on the any link node; The request identifier of the initial user request, the node identifier of the arbitrary link node, and the probe identifier of the probe preset on the arbitrary link node are associated and stored to generate a corresponding association information group.

3. The method according to claim 1, characterized in that The link messages are converted and processed respectively to obtain corresponding link tags, wherein a link tag is obtained by performing the following operations: Aggregate at least one associated information group included in the link message to obtain at least one aggregated sub-information; Performing splicing processing on the at least one aggregated sub-information to obtain target aggregated information; Mapping processing is performed on the target aggregation information to obtain a corresponding link label.

4. The method according to claim 3, characterized in that The aggregating at least one associated information group included in the link message to obtain at least one aggregated sub-information includes: Aggregating the associated information groups containing the same node identifier in the one link message to obtain the at least one aggregated sub-information; Each aggregated sub-information includes a node identifier and at least one probe identifier.

5. The method according to claim 4, characterized in that If one aggregation sub-information contains multiple probe identifiers, the probe identifiers in the one aggregation sub-information are sorted according to the order of the size of the probe identifiers.

6. The method according to claim 1, characterized in that Determine the candidate user request set for the current round by performing the following operations: If it is the first cycle, the initial user request set is used as the candidate user request set for the current round; If it is not the first cycle, the candidate user request set after deduplication in the previous round is used as the candidate user request set in the current round.

7. The method according to claim 1, characterized in that Determine the candidate user request set of a type in the current round by performing the following operations: If it is the first cycle, an initial user request set of a type is used as a candidate user request set for the current round; If it is not the first cycle, a candidate user request set after deduplication in the previous round is used as a candidate user request set in the current round.

8. The method according to claim 1, characterized in that The link label similarity between the two link labels is obtained by performing the following operations: Using a character deduplication algorithm, compare characters at the same position on the two link tags one by one to obtain the similarity of the link tags; or, A hash algorithm is used to obtain hash values ​​of the two link labels respectively, and Hamming distance processing is performed on the two hash values ​​to obtain the link label similarity.

9. A device for deduplicating massive requests, characterized in that: include: An acquisition unit, used to acquire a link message of each initial user request in the initial user request set, each link message being generated based on a link node providing services for the same initial user request; A deduplication unit, used to convert and process each link message respectively to obtain a corresponding link tag; In a cyclic manner, multiple rounds of deduplication are performed on an initial user request set to obtain a target user request set from which all duplicate requests have been removed, wherein multiple user request pairs are obtained by permutation and combination of the candidate user request set of the current round, and the following operations are performed respectively: when the link tag similarity between two candidate user requests in the user request pair exceeds a set threshold, any candidate user request is removed, and the candidate user request set of the current round is the initial user request set or the candidate user request set after deduplication in the previous round; Alternatively, a cyclic manner is adopted to perform multiple rounds of deduplication on candidate user request sets of different types, and obtain multiple target user request sets from which all duplicate requests have been eliminated, wherein multiple user request pairs are obtained by permuting and combining candidate user request sets of one type in the current round, and the following operations are performed respectively: when the link label similarity between two candidate user requests in the user request pair exceeds a set threshold, any candidate user request is eliminated, and the candidate user request set of one type in the current round is an initial user request set of one type or a candidate user request set of one type after deduplication in the previous round.

10. The device according to claim 9, characterized in that A link message includes at least one associated information group, wherein the device further includes a generating unit, and the generating unit generates an associated information group by performing the following operations: In response to monitoring an operation that any link node provides a service for an initial user request, obtaining at least a request identifier of the initial user request, a node identifier of the any link node, and a probe identifier of a probe preset on the any link node; The request identifier of the initial user request, the node identifier of the arbitrary link node, and the probe identifier of the probe preset on the arbitrary link node are associated and stored to generate a corresponding association information group.

11. The device according to claim 9, characterized in that The deduplication unit obtains a link tag by performing the following operations: Aggregate at least one associated information group included in the link message to obtain at least one aggregated sub-information; Performing splicing processing on the at least one aggregated sub-information to obtain target aggregated information; Mapping processing is performed on the target aggregation information to obtain a corresponding link label.

12. The device according to claim 11, characterized in that The deduplication unit is used for: Aggregating the associated information groups containing the same node identifier in the one link message to obtain the at least one aggregated sub-information; Each aggregated sub-information includes a node identifier and at least one probe identifier.

13. The device according to claim 12, characterized in that If one aggregate sub-information contains multiple probe identifiers, the probe identifiers in the one aggregate sub-information are sorted according to the order of the size of the probe identifiers.

14. The device according to claim 9, characterized in that The deduplication unit determines the candidate user request set of the current round by performing the following operations: If it is the first cycle, the initial user request set is used as the candidate user request set for the current round; If it is not the first cycle, the candidate user request set after deduplication in the previous round is used as the candidate user request set in the current round.

15. The device according to claim 9, characterized in that The deduplication unit determines a set of candidate user requests of a type in the current round by performing the following operations: If it is the first cycle, an initial user request set of a type is used as a candidate user request set for the current round; If it is not the first cycle, a candidate user request set after deduplication in the previous round is used as a candidate user request set in the current round.

16. The device according to claim 9, characterized in that The deduplication unit obtains the link label similarity between the two link labels by performing the following operations: Using a character deduplication algorithm, compare characters at the same position on the two link tags one by one to obtain the similarity of the link tags; or, A hash algorithm is used to obtain hash values ​​of the two link labels respectively, and Hamming distance processing is performed on the two hash values ​​to obtain the link label similarity.

17. A computer device, characterized in that: The method comprises a processor and a memory, wherein the memory stores program codes, and when the program codes are executed by the processor, the processor executes the steps of the method according to any one of claims 1 to 8.

18. A computer-readable storage medium, characterized in that: The method comprises a program code, and when the program code is run on a computer device, the program code is used to make the computer device execute the steps of the method according to any one of claims 1 to 8.

Images