A method and device for detecting attacks on network data automatic acquirer

Abstract

One or more embodiments of the present specification provide a method and device for detecting attacks by a network data auto-acquirer, the method being applied to a server, the server maintaining an alarm threshold, the method comprising: obtaining a first traffic sample set received by the server in a first time period and a second traffic sample set received in a second time period; calculating the characteristic distances of the first traffic sample set and the second traffic sample set respectively in at least one characteristic dimension, and determining a traffic anomaly index based on the characteristic distances in the at least one characteristic dimension; generating an attenuation factor, and adjusting the traffic anomaly index according to the attenuation factor, the attenuation factor being positively correlated to the number of samples in the first traffic sample set and / or the number of samples in the second traffic sample set; and determining that a network data auto-acquirer attack exists in the first time period or the second time period when the adjusted traffic anomaly index exceeds the alarm threshold.

Description

Technical Field

[0001] One or more embodiments of the present specification relate to the field of Internet technology, and in particular, to a method and device for detecting attacks on an automatic network data acquirer. Background Art

[0002] A network data automatic acquirer, also known as a web crawler, is a program or script that automatically grabs Internet information according to certain rules. However, the proliferation of web crawlers will constitute a crawler attack, causing the server to bear a large number of access requests, consuming server resources, and hindering normal user access. Therefore, how to detect crawler attacks is a technical problem that needs to be solved urgently.

[0003] On the one hand, the detection of traditional crawler attacks focuses on the characteristics of access traffic at the message level or session level, such as detecting the access frequency of IP addresses, detecting the request header format, detecting cookies (data stored on the user's local terminal), etc. However, these are all rule-based detection methods that first summarize the static characteristics of crawler traffic and then determine whether the received specific traffic meets these static characteristics. Although it is possible to identify whether a crawler attack has occurred from a rule perspective, it is easy to miss detections due to the lack of a holistic perspective and the difficulty in exhausting the static characteristics of crawler traffic. On the other hand, although it is also possible to detect whether a crawler attack may have occurred by detecting traffic fluctuations, the traditional method of detecting traffic fluctuations cannot distinguish whether the traffic fluctuations are caused by crawler attacks or normal access. Therefore, simply attributing the phenomenon of traffic fluctuations to crawler attacks will easily lead to false detections. Summary of the invention

[0004] In view of this, one or more embodiments of the present specification provide a method and device for detecting attacks on an automatic network data acquirer.

[0005] To achieve the above objectives, one or more embodiments of this specification provide the following technical solutions:

[0006] According to a first aspect of one or more embodiments of this specification, a method for detecting attacks on a network data automatic acquirer is proposed, which is applied to a server, wherein the server maintains an alarm threshold, and the method includes:

[0007] Acquire a first traffic sample set received by the server in a first time period and a second traffic sample set received in a second time period;

[0008] Calculating the characteristic distances of the first traffic sample set and the second traffic sample set in at least one characteristic dimension, and determining a traffic anomaly index based on the characteristic distances in the at least one characteristic dimension;

[0009] Generate an attenuation factor, and adjust the flow anomaly index according to the attenuation factor, wherein the attenuation factor is positively correlated to the number of samples in the first flow sample set and / or the number of samples in the second flow sample set;

[0010] When the adjusted traffic anomaly index exceeds the alarm threshold, it is determined that a network data automatic acquirer attack exists in the first time period or the second time period.

[0011] Optionally, the calculating the characteristic distances of the first traffic sample set and the second traffic sample set in at least one characteristic dimension respectively includes:

[0012] Based on the classification strategy corresponding to any one of the at least one characteristic dimension, classify the traffic samples included in the first traffic sample set, obtain the proportion of the traffic samples included in the first traffic sample set in each category, and generate the sample proportion distribution of the first traffic sample set under any one of the characteristic dimensions;

[0013] Based on the classification strategy corresponding to any of the feature dimensions, the traffic samples included in the second traffic sample set are classified, the proportion of the traffic samples included in the second traffic sample set in each category is obtained, and the sample proportion distribution of the second traffic sample set under any of the feature dimensions is generated accordingly;

[0014] The group stability index between the sample proportion distribution of the first traffic sample set in any characteristic dimension and the sample proportion distribution of the second traffic sample set in any characteristic dimension is determined as the characteristic distance between the first traffic sample set and the second traffic sample set in any characteristic dimension.

[0015] Optionally, the classification strategy corresponding to any feature dimension includes:

[0016] The traffic samples included in the traffic sample set to be classified are classified according to the preset categories included in any of the feature dimensions, and the classified categories are sorted in descending order according to the number of samples included, and all categories ranked below the preset ranking are merged into the same category.

[0017] Optionally, generating the attenuation factor includes:

[0018] Determine a first attenuation factor and a second attenuation factor based on the number of samples of the first flow sample set and the number of samples of the second flow sample set, respectively, wherein the first attenuation factor is positively correlated to the number of samples of the first flow sample set, and the second attenuation factor is positively correlated to the number of samples of the second flow sample set;

[0019] The attenuation factor is generated based on the first attenuation factor and the second attenuation factor.

[0020] Optionally, determining the first attenuation factor and the second attenuation factor based on the number of samples of the first traffic sample set and the number of samples of the second traffic sample set, respectively, includes:

[0021] After the number of samples of the first traffic sample set is input into the attenuation factor generation function, the larger one of the first trend attention factor and the output value of the attenuation factor generation function is determined as the first attenuation factor;

[0022] After the number of samples of the second traffic sample set is input into the attenuation factor generation function, the larger one of the second trend attention factor and the output value of the attenuation factor generation function is determined as the second attenuation factor;

[0023] The generating the attenuation factor based on the first attenuation factor and the second attenuation factor comprises:

[0024] The attenuation factor is determined as a product of the first attenuation factor and the second attenuation factor.

[0025] Optionally, any one of the first trend attention factor and the second trend attention factor is zero or close to zero, the other one of the first trend attention factor and the second trend attention factor is greater than zero, and the ratio of the other trend attention factor to any one of the trend attention factors exceeds a preset value.

[0026] Optionally, the server maintains a truncation threshold corresponding to a characteristic distance in at least two characteristic dimensions; the calculating the characteristic distances of the first traffic sample set and the second traffic sample set in at least one characteristic dimension, respectively, and determining the traffic anomaly index based on the characteristic distance in at least one characteristic dimension, includes:

[0027] Calculate at least two feature distances of the first traffic sample set and the second traffic sample set in the at least two feature dimensions respectively;

[0028] In the case where any characteristic distance among the at least two characteristic distances exceeds the truncation threshold corresponding to the any characteristic distance, re-determining the value of the any characteristic distance as the truncation threshold corresponding to the any characteristic distance;

[0029] The flow anomaly index is determined based on the re-determined at least two characteristic distances.

[0030] Optionally, the truncation threshold corresponding to the feature distance in any feature dimension maintained by the server is positively correlated to: the correlation between the one-dimensional traffic anomaly index of the historical traffic samples received by the server in any feature dimension and the attack of the historical network data automatic acquirer.

[0031] Optionally, determining the flow anomaly index based on the re-determined at least two characteristic distances includes:

[0032] Determine the weighted sum of the at least two re-determined characteristic distances as the flow anomaly index; or,

[0033] The quotient of the weighted sum of the re-determined at least two characteristic distances and the weighted sum of the truncation thresholds corresponding to the characteristic distances in the at least two characteristic dimensions is determined as the traffic anomaly index.

[0034] Optionally, the determining that there is a network data automatic acquirer attack in the first time period or the second time period includes:

[0035] When the number of samples in any one of the first traffic sample set and the second traffic sample set is greater than the number of samples in the other traffic sample set by a preset number, it is determined that a network data automatic acquirer attack exists on the server within a time period when the server receives any one of the traffic sample sets.

[0036] Optionally, also include:

[0037] Before calculating the characteristic distances of the first traffic sample set and the second traffic sample set respectively under the at least one characteristic dimension, the identified network data automatic acquirer traffic samples included in the first traffic sample set and the second traffic sample set are deleted.

[0038] Optionally, the first time period and the second time period are the same time period, the traffic samples included in the first traffic sample set are in either an anonymous state or a logged-in state, and the traffic samples included in the second traffic sample set are in the other of the anonymous state and the logged-in state; or,

[0039] The first time period and the second time period are adjacent time periods, and the traffic samples included in the first traffic sample set and the second traffic sample set are both in an anonymous state or a logged-in state.

[0040] Optionally, the at least one feature dimension includes at least one of the following: visitor's operating system, visitor's operating system version, visitor's browser, visitor's browser version, visitor's client version, visitor's GPS location, visitor's IP map location, traffic access duration, and cookie name.

[0041] According to a second aspect of one or more embodiments of this specification, a network data automatic acquirer attack detection device is proposed, which is applied to a server, and the server maintains an alarm threshold. The device includes:

[0042] A traffic sample set acquisition unit, used to acquire a first traffic sample set received by the server in a first time period and a second traffic sample set received in a second time period;

[0043] A flow anomaly index determination unit, configured to calculate a characteristic distance between the first flow sample set and the second flow sample set in at least one characteristic dimension, and determine a flow anomaly index based on the characteristic distance in the at least one characteristic dimension;

[0044] A flow anomaly index adjustment unit, used to generate an attenuation factor and adjust the flow anomaly index according to the attenuation factor, wherein the attenuation factor is positively correlated to the number of samples in the first flow sample set and / or the number of samples in the second flow sample set;

[0045] The network data automatic acquirer attack determination unit is used to determine that a network data automatic acquirer attack exists in the first time period or the second time period when the adjusted traffic anomaly index exceeds the alarm threshold.

[0046] Optionally, the traffic anomaly index determination unit is specifically used to:

[0047] Based on the classification strategy corresponding to any one of the at least one characteristic dimension, classify the traffic samples included in the first traffic sample set, obtain the proportion of the traffic samples included in the first traffic sample set in each category, and generate the sample proportion distribution of the first traffic sample set under any one of the characteristic dimensions;

[0048] Based on the classification strategy corresponding to any of the feature dimensions, the traffic samples included in the second traffic sample set are classified, the proportion of the traffic samples included in the second traffic sample set in each category is obtained, and the sample proportion distribution of the second traffic sample set under any of the feature dimensions is generated accordingly;

[0049] The group stability index between the sample proportion distribution of the first traffic sample set in any characteristic dimension and the sample proportion distribution of the second traffic sample set in any characteristic dimension is determined as the characteristic distance between the first traffic sample set and the second traffic sample set in any characteristic dimension.

[0050] Optionally, the classification strategy corresponding to any feature dimension includes:

[0051] The traffic samples included in the traffic sample set to be classified are classified according to the preset categories included in any of the feature dimensions, and the classified categories are sorted in descending order according to the number of samples included, and all categories ranked below the preset ranking are merged into the same category.

[0052] Optionally, the traffic anomaly index adjustment unit is specifically used to:

[0053] Determine a first attenuation factor and a second attenuation factor based on the number of samples of the first flow sample set and the number of samples of the second flow sample set, respectively, wherein the first attenuation factor is positively correlated to the number of samples of the first flow sample set, and the second attenuation factor is positively correlated to the number of samples of the second flow sample set;

[0054] The attenuation factor is generated based on the first attenuation factor and the second attenuation factor.

[0055] Optionally, the traffic anomaly index adjustment unit is further used to:

[0056] After the number of samples of the first traffic sample set is input into the attenuation factor generation function, the larger one of the first trend attention factor and the output value of the attenuation factor generation function is determined as the first attenuation factor;

[0057] After the number of samples of the second traffic sample set is input into the attenuation factor generation function, the larger one of the second trend attention factor and the output value of the attenuation factor generation function is determined as the second attenuation factor;

[0058] The attenuation factor is determined as a product of the first attenuation factor and the second attenuation factor.

[0059] Optionally, any one of the first trend attention factor and the second trend attention factor is zero or close to zero, the other one of the first trend attention factor and the second trend attention factor is greater than zero, and the ratio of the other trend attention factor to any one of the trend attention factors exceeds a preset value.

[0060] Optionally, the server maintains a truncation threshold corresponding to the feature distance in at least two feature dimensions; the traffic anomaly index determination unit is specifically used to:

[0061] Calculate at least two feature distances of the first traffic sample set and the second traffic sample set in the at least two feature dimensions respectively;

[0062] In the case where any characteristic distance among the at least two characteristic distances exceeds the truncation threshold corresponding to the any characteristic distance, re-determining the value of the any characteristic distance as the truncation threshold corresponding to the any characteristic distance;

[0063] The flow anomaly index is determined based on the re-determined at least two characteristic distances.

[0064] Optionally, the truncation threshold corresponding to the feature distance in any feature dimension maintained by the server is positively correlated to: the correlation between the one-dimensional traffic anomaly index of the historical traffic samples received by the server in any feature dimension and the attack of the historical network data automatic acquirer.

[0065] Optionally, the traffic anomaly index determination unit is further used to:

[0066] Determine the weighted sum of the at least two re-determined characteristic distances as the flow anomaly index; or,

[0067] The quotient of the weighted sum of the re-determined at least two characteristic distances and the weighted sum of the truncation thresholds corresponding to the characteristic distances in the at least two characteristic dimensions is determined as the traffic anomaly index.

[0068] Optionally, the network data automatic acquirer attack determination unit is specifically used to:

[0069] When the number of samples in any one of the first traffic sample set and the second traffic sample set is greater than the number of samples in the other traffic sample set by a preset number, it is determined that a network data automatic acquirer attack exists on the server within a time period when the server receives any one of the traffic sample sets.

[0070] Optionally, also include:

[0071] The traffic sample deletion unit is used to delete the identified network data automatic acquirer traffic samples contained in the first traffic sample set and the second traffic sample set before calculating the characteristic distances of the first traffic sample set and the second traffic sample set respectively under the at least one characteristic dimension.

[0072] Optionally, the first time period and the second time period are the same time period, the traffic samples included in the first traffic sample set are in either an anonymous state or a logged-in state, and the traffic samples included in the second traffic sample set are in the other of the anonymous state and the logged-in state; or,

[0073] The first time period and the second time period are adjacent time periods, and the traffic samples included in the first traffic sample set and the second traffic sample set are both in an anonymous state or a logged-in state.

[0074] Optionally, the at least one feature dimension includes at least one of the following: visitor's operating system, visitor's operating system version, visitor's browser, visitor's browser version, visitor's client version, visitor's GPS location, visitor's IP map location, traffic access duration, and cookie name.

[0075] According to a third aspect of one or more embodiments of this specification, an electronic device is provided, including:

[0076] processor;

[0077] a memory for storing processor-executable instructions;

[0078] The processor executes the executable instructions to implement the method for detecting attacks on the network data automatic acquirer as described in any of the above embodiments.

[0079] According to a fourth aspect of one or more embodiments of the present specification, there is provided a computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, implement the steps of the method for detecting attacks on a network data automatic acquirer as described in any of the above embodiments.

[0080] It can be seen from the above technical scheme that by calculating at least one characteristic distance between the first traffic sample set received in the first time period and the second traffic sample set received in the second time period in at least one characteristic dimension, the fluctuation of the overall stability of the server in at least one characteristic dimension can be collected, so as to identify the network data automatic acquisition attack according to at least one characteristic distance as the dynamic feature of the network data automatic acquisition attack, thereby improving the recognition accuracy of the network data automatic acquisition attack; at the same time, by adjusting the traffic anomaly index according to the attenuation factor generated according to the number of samples of the first traffic sample set and / or the number of samples of the second traffic sample set, the characterization ability of the traffic anomaly index with low credibility to characterize the activity intensity of the network data automatic acquisition is reduced, so that the value of the traffic anomaly index with lower credibility is also smaller, thereby avoiding false detection caused by too few samples as much as possible. BRIEF DESCRIPTION OF THE DRAWINGS

[0081] Figure 1 It is a flowchart of a method for detecting attacks on a network data automatic acquirer provided by an exemplary embodiment.

[0082] Figure 2 It is a schematic diagram of sample ratio distribution of a traffic sample set provided by an exemplary embodiment.

[0083] Figure 3 It is a structural schematic diagram of a device provided by an exemplary embodiment.

[0084] Figure 4 It is a block diagram of a network data automatic acquirer attack detection device provided by an exemplary embodiment. DETAILED DESCRIPTION

[0085] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with one or more embodiments of this specification. Instead, they are merely examples of devices and methods consistent with some aspects of one or more embodiments of this specification as detailed in the appended claims.

[0086] It should be noted that: in other embodiments, the steps of the corresponding method are not necessarily performed in the order shown and described in this specification. In some other embodiments, the steps included in the method may be more or less than those described in this specification. In addition, a single step described in this specification may be decomposed into multiple steps for description in other embodiments; and multiple steps described in this specification may be combined into a single step for description in other embodiments.

[0087] See also Figure 1 , Figure 1 A method for detecting attacks on a network data automatic acquirer provided by an exemplary embodiment is applied to a server, wherein the server maintains an alarm threshold, and the method comprises:

[0088] S102: Acquire a first traffic sample set received by the server in a first time period and a second traffic sample set received by the server in a second time period.

[0089] In the embodiments of this specification, the server will record the access traffic received in history and its access time, so as to be able to sort out the changes in the historical access traffic over time. For example, the server can record the access traffic received every day in the past according to the statistical unit of day. The access traffic involved in the embodiments of this specification can be understood as the access request from the outside received by the server interface. Each access request often carries relevant information about the access source, such as IP address, GPS location information, access party operating system, etc. This information will serve as the basis for the subsequent generation of sample features.

[0090] The first time period or the second time period involved in the embodiments of this specification belongs to a historical time period relative to the past. There is no requirement for a clear order between the first time period and the second time period, nor is there a need to maintain continuity in time (the first time period and the second time period can be spliced ​​together to form a complete time period), and the duration of the two can be equal or unequal. For example, the first time period can be within the past 24 hours, and the second time period can be all the time before the past 24 hours. Obviously, the duration of the first time period is usually much shorter than the second time period; or, the first time period can be January of last year, and the second time period can be January of this year. Of course, the first time period and the second time period can also be exactly the same, but in this case, there will be differences in the way the first traffic sample set and the second traffic sample set are obtained.

[0091] In an embodiment of the present specification, the first traffic sample set may include all access traffic received by the server in the first time period, or may be a partial traffic sample obtained by random sampling from all access traffic received in the first time period. Similarly, the second traffic sample set may also include all access traffic received by the server in the second time period, or a traffic sample obtained by random sampling from all access traffic received in the second time period. When the traffic samples included in the first traffic sample set or the second traffic sample set are obtained by sampling, the subsequent calculation scale of the feature distance calculated using the first traffic sample set and the second traffic sample set can be reduced, and computing resources can be saved by sacrificing confidence within a reasonable range. At the same time, the random sampling method can also ensure that the information is not distorted to the greatest extent, that is, the first traffic sample set or the second traffic sample set obtained by sampling can represent all access traffic received in the first time period or all access traffic received in the second time period as much as possible without losing credibility.

[0092] After the first flow sample set and the second flow sample set are collected, it is necessary to measure the difference between the first flow sample set and the second flow sample set in each feature dimension by calculating the feature distance, so as to judge the overall stability of the access flow received by the server in different time periods or different access states, and use this as a basis to identify whether the server has been attacked by crawlers. The access state involved in the embodiments of this specification refers to the login state of the access party when accessing the server interface (by accessing the site or through the app interface). If the access party is a registered user of the server platform and is already in the login state when accessing the server, the access state corresponding to its access flow is the login state, and if the access party is not registered or has not logged in when accessing the server, then the access state corresponding to its access flow is the anonymous state, which is used to indicate that the identity of the visitor is not a registered user.

[0093] In one embodiment, the first time period and the second time period may be the same time period, in which case the traffic samples included in the first traffic sample set are either the anonymous state or the logged-in state, and the traffic samples included in the second traffic sample set are the other of the anonymous state and the logged-in state. In the embodiment of this specification, although the first traffic sample set and the second traffic sample set are collected in the same time period, the sampling methods are different, so the traffic samples included in the different traffic sample sets obtained by the final sampling are also different. For example, the embodiment of this specification samples the access traffic of different access states in the same time period to obtain the first traffic sample set and the second traffic sample set, so the characteristic distance between the first traffic sample set and the second traffic obtained by the final calculation represents the overall difference between the traffic of different access states in the same time period.

[0094] In another embodiment, the first time period and the second time period are adjacent time periods, and the traffic samples contained in the first traffic sample set and the second traffic sample set are all in an anonymous state or a logged-in state. In the embodiment of the present specification, the first time period and the second time period do not overlap with each other but are adjacent to each other (i.e., they can be combined into a complete time period when spliced ​​together), for example, the first time period is yesterday, and the second time period is today. At this time, when sampling and obtaining the first traffic sample set and the second traffic sample set, traffic samples with the same access state will be selected, and the difference in sampling time periods will also cause the traffic samples contained in the different traffic sample sets finally sampled to be different, and the characteristic distance between the first traffic sample set and the second traffic that is finally calculated represents the overall difference between the traffic with the same access state in different consecutive time periods.

[0095] Based on the different sampling methods set above, the traffic samples contained in the first traffic sample set and the second traffic sample set finally obtained will have different attributes, and the subsequently calculated feature distances will also have different connotations, which will ultimately affect the type of crawler attack identified subsequently.

[0096] S104: Calculate the characteristic distances of the first traffic sample set and the second traffic sample set in at least one characteristic dimension, and determine a traffic anomaly index based on the characteristic distances in the at least one characteristic dimension.

[0097] In an embodiment of the present specification, the at least one feature dimension includes at least one of the following: visitor's operating system, visitor's operating system version, visitor's browser, visitor's browser version, visitor's client version, visitor's GPS location, visitor's IP map location, traffic access duration, and cookie name.

[0098] Optionally, the calculating the characteristic distances of the first traffic sample set and the second traffic sample set in at least one characteristic dimension respectively includes:

[0099] Based on the classification strategy corresponding to any one of the at least one characteristic dimension, classify the traffic samples included in the first traffic sample set, obtain the proportion of the traffic samples included in the first traffic sample set in each category, and generate the sample proportion distribution of the first traffic sample set under any one of the characteristic dimensions;

[0100] Based on the classification strategy corresponding to any of the feature dimensions, the traffic samples included in the second traffic sample set are classified, the proportion of the traffic samples included in the second traffic sample set in each category is obtained, and the sample proportion distribution of the second traffic sample set under any of the feature dimensions is generated accordingly;

[0101] The group stability index between the sample proportion distribution of the first traffic sample set in any characteristic dimension and the sample proportion distribution of the second traffic sample set in any characteristic dimension is determined as the characteristic distance between the first traffic sample set and the second traffic sample set in any characteristic dimension.

[0102] In the embodiments of the present specification, the classification strategy corresponding to any feature dimension is to classify according to the preset categories contained in any feature dimension. For example, for the feature dimension of the access party operating system, the preset categories contained therein may include Windows, Android, iOS, and Linux, etc., that is, instances of various different operating system types. Then, when the first traffic sample set or the second traffic sample set is classified based on the classification strategy corresponding to the feature dimension of the access party operating system, the number and proportion of the traffic samples contained in the first traffic sample set or the second traffic sample set are actually counted according to the different types of the access party operating system, so as to obtain the sample proportion distribution of the first traffic sample set or the second traffic sample set under the feature dimension of the access party operating system. The sample proportion distribution, as a distribution feature, contains the overall information of the original data set. Therefore, the overall difference between the first traffic sample set and the second traffic sample set can be judged by calculating the feature distance between the first traffic sample set and the second traffic sample set under the same feature dimension.

[0103] Figure 2 It is a schematic diagram of the sample ratio distribution of a traffic sample set provided by an exemplary embodiment. Assuming that the sample ratio distribution of the compared state is the sample ratio distribution of the first traffic sample set under the characteristic dimension of the access party operating system, and the sample ratio distribution of the comparison state is the sample ratio distribution of the second traffic sample set under the characteristic dimension of the access party operating system, and A, B, C, and D in the figure represent the four operating system types of Windows, Android, iOS, and Linux, respectively, it is obvious that the sample ratio distribution of the comparison state is somewhat different from the sample ratio distribution of the compared state, which is mainly reflected in that the proportion of access traffic with the access party operating system of Windows in the second traffic sample is relatively more than that in the first traffic sample, and the proportion of access traffic with the access party operating system of Linux in the second traffic sample is relatively less than that in the first traffic sample.

[0104] In order to describe the differences between different sample proportion distributions, it is necessary to calibrate the degree of difference between the comparison sample proportion distribution and the compared sample proportion distribution through the characteristic distance between the different sample proportion distributions. In the embodiment of the present specification, the group stability index between the sample proportion distribution of the first traffic sample set under any characteristic dimension and the sample proportion distribution of the second traffic sample set under any characteristic dimension is used as the characteristic distance to measure the difference between the first traffic sample set and the second traffic sample set under any characteristic dimension. Specifically, the group stability index between the sample proportion distribution of the first traffic sample set under any characteristic dimension and the sample proportion distribution of the second traffic sample set under any characteristic dimension is calculated in the following way: calculate the difference between the proportions of the first traffic sample set and the second traffic sample set in each of the same categories under the same characteristic dimension, and sum up all the obtained difference values.

[0105] Still Figure 2 For example, suppose that after the first traffic sample set and the second traffic sample set are classified according to the classification strategy corresponding to the characteristic dimension of the access party's operating system (a total of 4 categories: Windows, Android, iOS and Linux), 4 first traffic sample subsets as components of the first traffic sample set and another 4 second traffic sample subsets as components of the second traffic sample set are obtained, which correspond to the sets of traffic samples whose access party operating systems are Windows, Android, iOS and Linux in the first traffic sample set and the second traffic sample set, respectively. First, it is necessary to calculate the proportion of traffic samples contained in the first traffic sample set and the second traffic sample set in each category (that is, the proportion of each first traffic sample subset in the first traffic sample set and the proportion of each second traffic sample subset in the second traffic sample set), which can be calculated by the following formula:

[0106]

[0107] Among them, p c The proportion of traffic samples in the “c” category (Windows, Android, iOS, or Linux) in the corresponding sample proportion distribution, n c It is used to indicate the number of samples in the subset of traffic samples corresponding to the "c" category. C is used to indicate the number of preset categories included in the current classification strategy. Currently, C = 4. δ refers to a positive and small constant. The original intention of its setting is to prevent calculation errors (to prevent p c is 0), for example, δ can be 1e-200.

[0108] Then, it is necessary to calculate the population stability index between the sample proportion distribution of the first traffic sample set under any characteristic dimension and the sample proportion distribution of the second traffic sample set under any characteristic dimension, which can be calculated by the following formula:

[0109]

[0110] Where D is the characteristic distance between the first traffic sample and the second traffic sample in the characteristic dimension of the access party's operating system, PSI (Population Stability Index) is the population stability index between the sample proportion distribution of the first traffic sample set in the characteristic dimension of the access party's operating system and the sample proportion distribution of the second traffic sample set in the characteristic dimension of the access party's operating system, p i It is used to represent the proportion of traffic samples of category “i” (Windows, Android, iOS or Linux) in the distribution of the proportion of comparison samples, p i ' is used to indicate the proportion of traffic samples of category "i" in the proportion distribution of the compared samples, and C is used to indicate the number of preset categories included in the current classification strategy, and currently C = 4. It is easy to find that PSI satisfies symmetry, that is, if the proportion distribution of the comparison sample is swapped with the proportion distribution of the compared sample, the value obtained remains unchanged.

[0111] What is introduced above is a specific implementation method for calculating the characteristic distance between a traffic sample set and a second traffic sample set in any characteristic dimension. In the present application, it is necessary to calculate at least one characteristic distance between the first traffic sample set and the second traffic sample set respectively in the at least one characteristic dimension. Each characteristic distance corresponds to only one characteristic dimension, and each characteristic dimension also corresponds to only one characteristic distance.

[0112] Optionally, the classification strategy corresponding to any of the feature dimensions includes: classifying the traffic samples contained in the traffic sample set to be classified according to the preset categories contained in any of the feature dimensions, sorting the categories obtained by classification in order of the number of samples contained from high to low, and merging all categories ranked below the preset ranking into the same category. In the embodiments of this specification, since some feature dimensions contain too many categories, it will bring greater computing pressure to the subsequent sample proportion distribution and feature distance calculation. Therefore, in order to save computing resources and ensure distortion as much as possible, the multiple traffic sample subsets obtained by classification can be sorted according to the number of samples, and the categories corresponding to the top-ranked traffic sample subsets can be retained, and the categories corresponding to the bottom-ranked traffic sample subsets can be integrated into a new category, so as to simplify the categories contained in any of the feature dimensions, thereby limiting the total number of categories to an acceptable range. At the same time, retaining the sample categories with greater influence that rank at the top can maximize the guarantee that the simplified sample proportion distribution is still representative of the overall distribution characteristics of its traffic sample set, and will not cause significant distortion. In addition, by integrating multiple categories with low rankings into one category, the PSI value determined under the clustered feature dimension (the number of samples in different categories after classification is significantly different) can be relatively larger, while the PSI value determined under the dispersed feature dimension (the number of samples in different categories after classification is not significantly different) can be relatively smaller. This is because the proportion of the new categories integrated in the simplified sample proportion distribution obtained under the clustered feature dimension is small, but the proportion of the new categories integrated in the simplified sample proportion distribution obtained under the dispersed feature dimension is large. As a result, when PSI is calculated subsequently, the difference values ​​corresponding to the new categories have different contributions to PSI under feature dimensions of different natures.

[0113] In an embodiment of the present specification, the determination of the traffic anomaly index based on the characteristic distance under the at least one characteristic dimension includes: determining the weighted sum of the characteristic distances under the at least one characteristic dimension as the traffic anomaly index. For example, in the case where only the characteristic distance under one characteristic dimension is determined, the characteristic distance can be directly regarded as the traffic anomaly index; or, in the case where three characteristic distances under three characteristic dimensions are determined, the weight corresponding to each characteristic distance (that is, the server pre-maintains the weight corresponding to the characteristic distance under each characteristic dimension) characteristic distance can be first determined respectively, and then each characteristic distance is multiplied by the corresponding weight and the sum is obtained as the traffic anomaly index. Of course, in addition to the above-mentioned method of calculating the weighted sum of characteristic distances to determine the traffic anomaly index, other methods can also be used to obtain the traffic anomaly index, but it is necessary to ensure that the traffic anomaly index is positively correlated with each characteristic distance involved in determining the traffic anomaly index. This specification does not limit other specific methods for determining the traffic anomaly index.

[0114] S106: Generate an attenuation factor, and adjust the flow anomaly index according to the attenuation factor, wherein the attenuation factor is positively correlated with the number of samples in the first flow sample set and / or the number of samples in the second flow sample set.

[0115] In an embodiment of the present specification, if the number of samples collected in the first traffic sample set and / or the second traffic sample set is too small, the corresponding sample ratio distribution determined based on the first traffic sample set or the second traffic sample set will naturally be unstable and lack credibility. At this time, the characteristic distance calculated between the first traffic sample set and the second traffic sample set cannot accurately reflect that there is indeed a large distribution characteristic difference between the first traffic sample set and the second traffic sample set. In view of this, in an embodiment of the present specification, an attenuation factor is generated according to the number of samples in the first traffic sample set and / or the number of samples in the second traffic sample set, and then the attenuation factor is used to adjust the traffic anomaly index, thereby lowering the value of the traffic anomaly index with a lower confidence level, that is, it is inclined to believe that there is no crawler attack when the crawler attack cannot be accurately determined, thereby reducing the false detection rate to a certain extent.

[0116] The embodiments of this specification do not limit the specific method of generating the attenuation factor, but still need to limit the attenuation factor to be positively correlated with the number of samples of the first flow sample set and / or the number of samples of the second flow sample set, so as to ensure that the more samples there are in at least one flow sample set, the larger the attenuation factor. At the same time, the embodiments of this specification do not limit the specific method of adjusting the flow anomaly index according to the attenuation factor to obtain the adjusted flow anomaly index, but need to ensure that the adjusted flow anomaly index is positively correlated with the flow anomaly index and the attenuation factor before adjustment, and the adjusted flow anomaly index does not exceed the flow anomaly index before adjustment, so that the larger the attenuation factor, the smaller the attenuation adjustment effect on the flow anomaly index. For example, the product of the flow anomaly index and the attenuation factor can be determined as the adjusted flow anomaly index, and the value range of the attenuation factor is limited to between 0 and 1.

[0117] Optionally, generating the attenuation factor includes: determining a first attenuation factor and a second attenuation factor based on the number of samples in the first traffic sample set and the number of samples in the second traffic sample set, respectively, wherein the first attenuation factor is positively correlated with the number of samples in the first traffic sample set, and the second attenuation factor is positively correlated with the number of samples in the second traffic sample set; generating the attenuation factor based on the first attenuation factor and the second attenuation factor.

[0118] In the embodiment of the present specification, the generation of the attenuation factor requires reference to the sample numbers of the first flow sample set and the second flow sample set at the same time, and the adjustment factors corresponding to the sample numbers of the first flow sample set and the second flow sample set are defined by the first attenuation factor and the second attenuation factor, respectively, so that the first attenuation factor and the second attenuation factor are independent of each other, and the two jointly determine the attenuation factor. The first attenuation factor and the second attenuation factor can be obtained by, but not limited to, the following calculation formula:

[0119]

[0120] Among them, r j Used to represent the "j" attenuation factor (first attenuation factor or second attenuation factor), N j It is used to represent the number of samples of the "j" traffic sample set (the first traffic sample set or the second traffic sample set), f() is a decay factor generating function with the independent variable being the number of samples, sigmoid() is a monotonically increasing continuous function with a domain of (-∞, +∞) and a range of (-1,1), β j is the stable number corresponding to the “j” flow sample set. The larger its value is, the higher the number of samples in the “j” sample set is required to make r j Significant.

[0121] The embodiments of this specification do not limit the specific method of generating the attenuation factor based on the first attenuation factor and the second attenuation factor, but need to limit the attenuation factor to be positively correlated with the first attenuation factor and the second attenuation factor. For example, the product of the first attenuation factor and the second attenuation factor can be used as the attenuation factor, or the weighted sum of the first attenuation factor and the second attenuation factor can be determined as the attenuation factor.

[0122] Optionally, determining the first attenuation factor and the second attenuation factor based on the number of samples of the first traffic sample set and the number of samples of the second traffic sample set, respectively, includes:

[0123] After the number of samples of the first traffic sample set is input into the attenuation factor generation function, the larger one of the first trend attention factor and the output value of the attenuation factor generation function is determined as the first attenuation factor;

[0124] After the number of samples of the second traffic sample set is input into the attenuation factor generation function, the larger one of the second trend attention factor and the output value of the attenuation factor generation function is determined as the second attenuation factor;

[0125] The generating the attenuation factor based on the first attenuation factor and the second attenuation factor comprises:

[0126] The attenuation factor is determined as a product of the first attenuation factor and the second attenuation factor.

[0127] R=max[f(N1),γ1]*max[f(N2),γ2]

[0128] Among them, R is used to represent the attenuation factor, f() is an attenuation factor generating function with the independent variable being the number of samples (when the independent variable is non-negative, the dependent variable is between 0 and 1), N1 and N2 are used to represent the number of samples in the first traffic sample set and the number of samples in the second traffic sample set, respectively, γ1 and γ2 are used to represent the first trend attention factor and the second trend attention factor, respectively, and γ1 and γ2 are both between 0 and 1, thereby ultimately ensuring that R is between 0 and 1. In this way, when the product of the traffic anomaly index and the attenuation factor is determined as the adjusted traffic anomaly index, it can be ensured that the adjusted traffic anomaly index does not exceed the traffic anomaly index before adjustment.

[0129] In the embodiments of the present specification, it is not difficult to find that γ1 and γ2 can control the minimum values ​​of the first attenuation factor and the second attenuation factor respectively, and since the adjusted flow anomaly index is positively correlated with the first attenuation factor, the second attenuation factor and the flow anomaly index before adjustment, the attenuation effect of the flow anomaly index based on the number of samples can be effectively suppressed by setting the first trend attention factor and the second trend attention factor, so that no matter how small the number of samples is, a minimum attention will be given to a certain extent to avoid the adjusted flow anomaly index from decaying to near 0.

[0130] Optionally, any one of the first trend attention factor and the second trend attention factor is zero or close to zero, the other one of the first trend attention factor and the second trend attention factor is greater than zero, and the ratio of the other trend attention factor to the any one of the trend attention factors exceeds a preset value. For example, γ1=0.5 and γ2=0 can be set, so that when the number of samples in the first traffic sample set is small, at least 0.5 attention can still be obtained. If the first time period is forward adjacent to the second time period, at least 0.5 attention can be paid to the "from low to high" trend of traffic on the time axis. If the first time period and the second time period are the same time period but the first traffic sample set and the second traffic sample set collect traffic samples of different access states respectively, at least 0.5 attention can be paid to the distribution characteristics of anonymous state traffic or logged-in state traffic. Similarly, by setting the first trend attention factor and the second trend attention factor with significant differences, unconditional attention to different trends of traffic on the timeline or traffic with different access states in the same time period can be achieved, and the degree of attention can also be freely adjusted.

[0131] S108: When the adjusted traffic anomaly index exceeds the alarm threshold, determine that a network data automatic acquirer attack exists in the first time period or the second time period.

[0132] In the embodiment of the present specification, the server maintains an alarm threshold, and the server determines whether the service is under a crawler attack by judging whether the traffic anomaly index finally determined exceeds the alarm threshold. Since the traffic anomaly index objectively represents the change in the distribution characteristics of the traffic, it reflects the intensity of crawler activity in a sense, which means that the risk of being attacked by crawlers is positively correlated with the traffic anomaly index. Therefore, the alarm threshold can also be set in a hierarchical manner, that is, multiple alarm thresholds are set in order of high and low. When the traffic anomaly index exceeds the alarm thresholds of different levels, it means that the degree of harm of the crawler attack has reached different levels, and alarm programs of different urgency are triggered, thereby realizing the hierarchical alarm of crawler attacks, helping maintenance personnel to focus on crawler attacks with greater harm, and not be diverted by crawler attacks with less harm.

[0133] In an embodiment of the present specification, the determination of the presence of a network data automatic acquirer attack in the first time period or the second time period includes: when the number of samples of any of the first traffic sample set and the second traffic sample set is greater than the number of samples of the other traffic sample set by a preset number, determining that the server has a network data automatic acquirer attack in the time period when the server receives the any traffic sample set. Since a crawler attack is manifested as an increase in access traffic, it is possible to determine whether the difference between the number of samples of the first traffic sample set and the second traffic sample set reaches a preset threshold to determine whether a crawler attack exists in the time period when the server receives the traffic sample set with a larger number of samples. For example, when the number of samples of the first traffic sample set is 10,000 more than the number of samples of the second traffic sample set, and / or the number of samples of the first traffic sample set is more than 1.5 times the number of samples of the second traffic sample set, it can be determined that the server has suffered a crawler attack in the first time period. It should be pointed out that when it is determined by the above method that a crawler attack exists in any of the first time period and the second time period, it does not mean that a crawler attack does not exist in the other of the first time period and the second time period, and it may still be detected as a crawler attack in subsequent crawler attack detection.

[0134] It can be seen from the above technical scheme that by calculating at least one characteristic distance between the first traffic sample set received in the first time period and the second traffic sample set received in the second time period in at least one characteristic dimension, the fluctuation of the overall stability of the server in at least one characteristic dimension can be collected, so as to identify the crawler attack according to at least one characteristic distance which is a dynamic feature of the crawler attack, thereby improving the recognition accuracy of the crawler attack; at the same time, by adjusting the traffic anomaly index according to the attenuation factor generated according to the number of samples of the first traffic sample set and / or the number of samples of the second traffic sample set, the characterization ability of the traffic anomaly index with low credibility to characterize the intensity of crawler activity is reduced, so that the value of the traffic anomaly index with lower credibility is also smaller, thereby avoiding false detection caused by too few samples as much as possible.

[0135] As mentioned above, the difference in sampling methods for the first traffic sample set and the second traffic sample set will affect the type of crawler attack identified later. In one embodiment, the first time period and the second time period are adjacent time periods, and the traffic samples contained in the first traffic sample set and the second traffic sample set are all anonymous or logged-in. In the embodiment of this specification, it is assumed that without the interference of crawler attacks, the distribution characteristics (such as sample proportion distribution) of the traffic sample sets of the same access state received by the server in adjacent different time periods are similar in each feature dimension. Therefore, if the distribution characteristics of the same access state traffic in different time periods are greatly different (that is, the aforementioned calculated characteristic distance is large), it can be indicated that the server has suffered a crawler attack of the corresponding access state in the first time period or the second time period. Since the embodiment of this specification collects traffic sample sets in different adjacent time periods, the characteristic distance calculated in this case reflects the change in the distribution characteristics of the same access state traffic on the time axis, that is, in this case, only short-term crawler attacks (including anonymous short-term crawler attacks or logged-in short-term crawler attacks) that last in a short time period can be identified, and long-term crawler attacks that last for a long time cannot be identified.

[0136] In another embodiment, the first time period and the second time period may be the same time period, the traffic samples included in the first traffic sample set are in any one of the anonymous state and the logged-in state, and the traffic samples included in the second traffic sample set are in the other of the anonymous state and the logged-in state. In the embodiment of the present specification, it is assumed that in the absence of interference from crawler attacks, the distribution characteristics of the anonymous state and the logged-in state traffic received by the server in the same time period are similar in each feature dimension, and crawler attacks often appear in the form of anonymous state traffic. Therefore, if the distribution characteristics of anonymous state traffic are significantly different from those of logged-in state traffic in the same time period, it may indicate that the server has suffered an anonymous state crawler attack in the same time period. Therefore, through the sampling method provided in the embodiment of the present specification, long-term anonymous crawler attacks can be identified.

[0137] In yet another embodiment, at least two traffic anomaly indicators are calculated respectively based on at least two sampling methods including the above-mentioned multiple sampling methods, and then the larger one of the at least two traffic anomaly indicators is taken as the significant traffic anomaly indicator, and when the significant traffic anomaly index exceeds the alarm threshold, it is determined that there is a crawler attack in the first time period or the second time period, which is used to replace the direct use of the comparison between the traffic anomaly indicator and the alarm threshold in the aforementioned crawler attack detection scheme to identify the crawler attack, so as to achieve the effect of simultaneously detecting multiple different types of crawler attacks, and in this embodiment, as long as one type of crawler attack occurs, the alarm mechanism will be triggered, thereby further reducing the missed detection rate of crawler attacks.

[0138] Optionally, it also includes: before calculating the characteristic distances of the first traffic sample set and the second traffic sample set under the at least one characteristic dimension, deleting the identified network data automatic acquirer traffic samples contained in the first traffic sample set and the second traffic sample set. In the embodiment of the present specification, the server will record the crawler traffic that has been identified in the past and the corresponding discrimination rules, and this part of the crawler traffic may still be mixed in the first traffic sample set or the second traffic sample set. Therefore, the crawler traffic samples can be identified from the first traffic sample set and the second traffic sample set according to the discrimination rules (that is, detected by rule detection) and deleted, and then the subsequent calculation of the sample proportion distribution and characteristic distance is performed to avoid repeated identification of the crawler attack that has been identified, so that when the traffic anomaly index exceeds the alarm threshold, it can be determined that there is a new crawler attack that has not been discovered before in the first time period or the second time period.

[0139] Optionally, the server maintains a truncation threshold corresponding to a characteristic distance in at least two characteristic dimensions; the calculating the characteristic distances of the first traffic sample set and the second traffic sample set in at least one characteristic dimension, respectively, and determining the traffic anomaly index based on the characteristic distance in at least one characteristic dimension, includes:

[0140] Calculate at least two feature distances of the first traffic sample set and the second traffic sample set in the at least two feature dimensions respectively;

[0141] In the case where any characteristic distance among the at least two characteristic distances exceeds the truncation threshold corresponding to the any characteristic distance, re-determining the value of the any characteristic distance as the truncation threshold corresponding to the any characteristic distance;

[0142] The flow anomaly index is determined based on the re-determined at least two characteristic distances.

[0143] In an embodiment of the present specification, the server maintains truncation thresholds corresponding to feature distances in at least two feature dimensions. For example, the server may maintain a unified truncation threshold for the feature distances in the at least two feature dimensions, or the server may maintain different truncation thresholds corresponding to the feature distances in different feature dimensions.

[0144] For example, suppose the server determines two feature distances, namely, the feature distance D under the feature dimension of the IP map location of the access party x The characteristic distance D from the operating system of the access party under the characteristic dimension y , then the server will search for the cutoff threshold corresponding to the feature distance according to the feature dimension where the feature distance is located. For example, the server searches for the feature distance D x The cutoff threshold is T x , characteristic distance D y The cutoff threshold is T y , then the feature distances can be re-determined according to the following formula:

[0145] D′ k =min(T k ,D k )

[0146] Among them, D k It is used to represent the feature distance under the “k” feature dimension (the IP map location of the visitor or the operating system of the visitor), D k ' is used to represent the feature distance under the re-determined "k" feature dimension, T k It is used to represent the truncation threshold of the feature distance under the feature dimension "k". It is not difficult to find that when any feature distance exceeds the truncation threshold corresponding to the any feature distance, the value of the any feature distance will be re-determined as the truncation threshold corresponding to the any feature distance, and when the value of the any feature distance does not exceed the truncation threshold corresponding to the any feature distance, the value of the any feature distance will not be re-determined, or in other words, the value of the any feature distance will be re-determined as the value of the any feature distance.

[0147] Since the determination of the traffic anomaly index depends on at least two characteristic distances, and the truncation threshold can obviously limit the upper limit of the characteristic distance, so that each characteristic distance has an upper limit (contribution upper limit) in its influence on the traffic anomaly index. In addition, for some characteristic distances with lower truncation thresholds, it is easy to reach the corresponding contribution upper limit, so its linear change interval (the interval in which the value of the characteristic distance is not limited by the truncation threshold) is smaller than the linear change interval of the characteristic distance with a higher truncation threshold, which means that it has lower sensitivity during adjustment. By setting the truncation threshold, when at least two characteristic distances are involved in determining the traffic anomaly index, the characteristic distance with a high contribution upper limit will also have high sensitivity for adjustment, thereby improving the accuracy and validity of the final determined traffic anomaly index in characterizing the intensity of crawler activity.

[0148] In the embodiment of the present specification, the traffic anomaly index can be calculated by the at least two characteristic distances. Specifically, the traffic anomaly index is determined based on the at least two re-determined characteristic distances, including: determining the weighted sum of the at least two re-determined characteristic distances as the traffic anomaly index; or determining the quotient of the weighted sum of the at least two re-determined characteristic distances and the weighted sum of the truncation thresholds corresponding to the characteristic distances under the at least two characteristic dimensions as the traffic anomaly index. The server maintains the weights corresponding to the at least two characteristic dimensions for calculating the weighted sum of the at least two characteristic distances and the weighted sum of the truncation thresholds corresponding to the characteristic distances under the at least two characteristic dimensions.

[0149] For example, suppose the server re-determines two feature distances, namely the feature distance D under the feature dimension of the IP map location of the access party x 'The characteristic distance D under the characteristic dimension of the accessing operating system y ', then the calculation formula of the traffic anomaly index can be expressed as:

[0150]

[0151] Among them, Score is used to represent the traffic anomaly index, D k ' is used to represent the re-determined feature distance under the "k" feature dimension (the IP map position of the visitor or the operating system of the visitor), T k The cutoff threshold used to represent the feature distance in the feature dimension "k", a k It is used to represent the weight corresponding to the “k” feature dimension. K is used to represent the number of feature dimensions involved in determining the traffic anomaly index. Currently, K=2.

[0152] In the embodiments of this specification, a kThe settings can be unified or differentiated according to different feature dimensions. For example, when a k When performing differentiated settings, the weight corresponding to the feature distance under any feature dimension can be made positively correlated to: the correlation between the single-dimensional traffic anomaly index of the historical traffic sample received by the server under any feature dimension and the attack of the historical network data automatic acquirer; or, the weight corresponding to any feature dimension can also be set to the inverse of the truncation threshold of the feature distance under any feature dimension, that is, a k Set it according to the following formula:

[0153]

[0154] According to the above formula, k In the case of setting, it can ensure that the upper limit of contribution of each feature distance to the traffic anomaly index is the same, thereby improving the effect of the truncation threshold from acting on both the contribution upper limit and the adjustment sensitivity to acting only on the adjustment sensitivity. This allows the adjustment sensitivity of feature distances of different feature dimensions to be set separately as an independent dimension, which helps to refine the influencing factors of the traffic anomaly index and improve the accuracy of the traffic anomaly index in reflecting the intensity of crawler activity.

[0155] Optionally, the truncation threshold corresponding to the feature distance under any feature dimension maintained by the server is positively correlated to: the correlation between the single-dimensional traffic anomaly index of the historical traffic samples received by the server under any feature dimension and the historical network data automatic acquirer attack. In the embodiment of this specification, different truncation thresholds can be set for the feature distances under different feature dimensions based on historical experience, so that the truncation threshold of the feature distance under the feature dimension with higher recognition ability for crawler attacks is larger, that is, it has a higher contribution upper limit to determining the traffic anomaly index, thereby further improving the accuracy and validity of the traffic anomaly index in characterizing the intensity of crawler activity. Specifically, the traffic anomaly index mentioned above can be regarded as a traffic anomaly index under at least two feature dimensions, and the method for generating a one-dimensional traffic anomaly index under any feature dimension based on historical traffic samples is basically the same as the method for determining the traffic anomaly index under at least two feature distances involved in the present application. The only difference is that there is only one feature distance determined by the one-dimensional traffic anomaly index. Therefore, the one-dimensional traffic anomaly index finally determined is also obtained based on one feature distance. At the same time, its feature distance is obtained by splitting the time axis up to now into several continuous historical time periods of the same length, and then calculating the feature distance according to the server in each historical time period and its previous historical time period adjacent to it (the neighboring historical time period in The characteristic distance between the historical traffic sample set received before the historical time period and the previous historical traffic sample set in any characteristic dimension (or the characteristic distance between the historical traffic sample sets of different access states in each historical time period in any characteristic dimension) is calculated to determine the one-dimensional traffic anomaly index corresponding to each historical time period. On the other hand, since the intensity of crawler activity in each historical time period is also determined, the correlation between the one-dimensional traffic anomaly index corresponding to each historical time period and the intensity of crawler activity in the historical time period is calculated to finally determine the correlation between the one-dimensional traffic anomaly index of the historical traffic samples received by the server in any characteristic dimension and the historical crawler attack. The server can determine the correlation between the one-dimensional traffic anomaly index and the historical crawler attack in multiple characteristic dimensions in the above manner. Obviously, the characteristic distance in the characteristic dimension corresponding to the one-dimensional traffic anomaly index with a larger correlation has better recognition ability when identifying crawler attacks. Increasing the corresponding truncation threshold helps to improve the recognition accuracy of crawler attacks.

[0156] Figure 3 is a schematic structural diagram of a device provided by an exemplary embodiment. Figure 3At the hardware level, the device includes a processor 302, an internal bus 304, a network interface 306, a memory 308, and a non-volatile memory 310, and may also include hardware required for other functions. One or more embodiments of this specification may be implemented based on software, such as the processor 302 reading the corresponding computer program from the non-volatile memory 310 into the memory 308 and then running it. Of course, in addition to the software implementation, one or more embodiments of this specification do not exclude other implementations, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0157] Please refer to Figure 4 , the network data automatic acquisition device attack detection device can be applied to Figure 3 The device shown in the figure is used to implement the technical solution of this specification. The network data automatic acquisition device attack detection device is applied to a server, and the server maintains an alarm threshold. The device includes:

[0158] The traffic sample set acquisition unit 401 is used to acquire a first traffic sample set received by the server in a first time period and a second traffic sample set received by the server in a second time period.

[0159] The traffic anomaly index determination unit 402 is used to calculate the characteristic distance between the first traffic sample set and the second traffic sample set in at least one characteristic dimension, and determine the traffic anomaly index based on the characteristic distance in the at least one characteristic dimension.

[0160] The flow anomaly index adjustment unit 403 is used to generate an attenuation factor and adjust the flow anomaly index according to the attenuation factor, wherein the attenuation factor is positively correlated with the number of samples in the first flow sample set and / or the number of samples in the second flow sample set.

[0161] The network data automatic acquirer attack determining unit 404 is used to determine that a network data automatic acquirer attack exists in the first time period or the second time period when the adjusted traffic anomaly index exceeds the alarm threshold.

[0162] Optionally, the traffic anomaly index determining unit 402 is specifically configured to:

[0163] Based on the classification strategy corresponding to any one of the at least one characteristic dimension, classify the traffic samples included in the first traffic sample set, obtain the proportion of the traffic samples included in the first traffic sample set in each category, and generate the sample proportion distribution of the first traffic sample set under any one of the characteristic dimensions;

[0164] Based on the classification strategy corresponding to any of the feature dimensions, the traffic samples included in the second traffic sample set are classified, the proportion of the traffic samples included in the second traffic sample set in each category is obtained, and the sample proportion distribution of the second traffic sample set under any of the feature dimensions is generated accordingly;

[0165] The group stability index between the sample proportion distribution of the first traffic sample set in any characteristic dimension and the sample proportion distribution of the second traffic sample set in any characteristic dimension is determined as the characteristic distance between the first traffic sample set and the second traffic sample set in any characteristic dimension.

[0166] Optionally, the classification strategy corresponding to any feature dimension includes:

[0167] The traffic samples included in the traffic sample set to be classified are classified according to the preset categories included in any of the feature dimensions, and the classified categories are sorted in descending order according to the number of samples included, and all categories ranked below the preset ranking are merged into the same category.

[0168] Optionally, the traffic anomaly index adjustment unit 403 is specifically used to:

[0169] Determine a first attenuation factor and a second attenuation factor based on the number of samples of the first flow sample set and the number of samples of the second flow sample set, respectively, wherein the first attenuation factor is positively correlated to the number of samples of the first flow sample set, and the second attenuation factor is positively correlated to the number of samples of the second flow sample set;

[0170] The attenuation factor is generated based on the first attenuation factor and the second attenuation factor.

[0171] Optionally, the traffic anomaly index adjustment unit 403 is further configured to:

[0172] After the number of samples of the first traffic sample set is input into the attenuation factor generation function, the larger one of the first trend attention factor and the output value of the attenuation factor generation function is determined as the first attenuation factor;

[0173] After the number of samples of the second traffic sample set is input into the attenuation factor generation function, the larger one of the second trend attention factor and the output value of the attenuation factor generation function is determined as the second attenuation factor;

[0174] The attenuation factor is determined as a product of the first attenuation factor and the second attenuation factor.

[0175] Optionally, any one of the first trend attention factor and the second trend attention factor is zero or close to zero, the other one of the first trend attention factor and the second trend attention factor is greater than zero, and the ratio of the other trend attention factor to any one of the trend attention factors exceeds a preset value.

[0176] Optionally, the server maintains a truncation threshold corresponding to the feature distance in at least two feature dimensions; the traffic anomaly index determination unit 402 is specifically used to:

[0177] Calculate at least two feature distances of the first traffic sample set and the second traffic sample set in the at least two feature dimensions respectively;

[0178] In the case where any characteristic distance among the at least two characteristic distances exceeds the truncation threshold corresponding to the any characteristic distance, re-determining the value of the any characteristic distance as the truncation threshold corresponding to the any characteristic distance;

[0179] The flow anomaly index is determined based on the re-determined at least two characteristic distances.

[0180] Optionally, the truncation threshold corresponding to the feature distance in any feature dimension maintained by the server is positively correlated to: the correlation between the one-dimensional traffic anomaly index of the historical traffic samples received by the server in any feature dimension and the attack of the historical network data automatic acquirer.

[0181] Optionally, the traffic anomaly index determining unit 402 is further configured to:

[0182] Determine the weighted sum of the at least two re-determined characteristic distances as the flow anomaly index; or,

[0183] The quotient of the weighted sum of the re-determined at least two characteristic distances and the weighted sum of the truncation thresholds corresponding to the characteristic distances in the at least two characteristic dimensions is determined as the traffic anomaly index.

[0184] Optionally, the network data automatic acquirer attack determination unit 404 is specifically used to:

[0185] When the number of samples in any one of the first traffic sample set and the second traffic sample set is greater than the number of samples in the other traffic sample set by a preset number, it is determined that a network data automatic acquirer attack exists on the server within a time period when the server receives any one of the traffic sample sets.

[0186] Optionally, also include:

[0187] The traffic sample deletion unit 405 is used to delete the identified network data automatic acquirer traffic samples contained in the first traffic sample set and the second traffic sample set before calculating the characteristic distances of the first traffic sample set and the second traffic sample set respectively under the at least one characteristic dimension.

[0188] Optionally, the first time period and the second time period are the same time period, the traffic samples included in the first traffic sample set are in either an anonymous state or a logged-in state, and the traffic samples included in the second traffic sample set are in the other of the anonymous state and the logged-in state; or,

[0189] The first time period and the second time period are adjacent time periods, and the traffic samples included in the first traffic sample set and the second traffic sample set are both in an anonymous state or a logged-in state.

[0190] Optionally, the at least one feature dimension includes at least one of the following: visitor's operating system, visitor's operating system version, visitor's browser, visitor's browser version, visitor's client version, visitor's GPS location, visitor's IP map location, traffic access duration, and cookie name.

[0191] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, which may be in the form of a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email transceiver, a game console, a tablet computer, a wearable device or a combination of any of these devices.

[0192] In a typical configuration, a computer includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0193] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0194] Computer-readable media include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, disk storage, quantum memory, graphene-based storage media or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include temporary computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0195] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0196] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0197] The terms used in one or more embodiments of this specification are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of this specification. The singular forms of "a", "said" and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0198] It should be understood that although the terms first, second, third, etc. may be used to describe various information in one or more embodiments of this specification, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0199] The above description is merely a preferred embodiment of one or more embodiments of the present specification and is not intended to limit one or more embodiments of the present specification. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of one or more embodiments of the present specification shall be included in the scope of protection of one or more embodiments of the present specification.

Claims

1. A method for detecting attacks on a network data automatic acquirer, applied to a server, wherein the server maintains an alarm threshold, the method comprising: Acquire a first traffic sample set received by the server in a first time period and a second traffic sample set received in a second time period; Calculating the characteristic distances of the first traffic sample set and the second traffic sample set in at least one characteristic dimension, and determining a traffic anomaly index based on the characteristic distances in the at least one characteristic dimension; The method of calculating the characteristic distance between the first traffic sample set and the second traffic sample set in at least one characteristic dimension respectively comprises: determining the characteristic distance between the first traffic sample set and the second traffic sample set in any characteristic dimension by a group stability index between the distribution characteristics of the first traffic sample set in any characteristic dimension and the distribution characteristics of the second traffic sample set in any characteristic dimension; the group stability index is obtained by the following calculation method: calculating the difference between the proportions of the first traffic sample set and the second traffic sample set in each same category in the same characteristic dimension, and summing up all the obtained difference values; Generate an attenuation factor, and adjust the flow anomaly index according to the attenuation factor, wherein the attenuation factor is positively correlated to the number of samples in the first flow sample set and / or the number of samples in the second flow sample set; When the adjusted traffic anomaly index exceeds the alarm threshold, it is determined that a network data automatic acquirer attack exists in the first time period or the second time period.

2. According to the method of claim 1, determining the group stability index between the distribution characteristics of the first traffic sample set in any characteristic dimension and the distribution characteristics of the second traffic sample set in any characteristic dimension as the characteristic distance between the first traffic sample set and the second traffic sample set in any characteristic dimension comprises: Based on the classification strategy corresponding to any one of the at least one characteristic dimension, classify the traffic samples included in the first traffic sample set, obtain the proportion of the traffic samples included in the first traffic sample set in each category, and generate the sample proportion distribution of the first traffic sample set under any one of the characteristic dimensions; Based on the classification strategy corresponding to any of the feature dimensions, the traffic samples included in the second traffic sample set are classified, the proportion of the traffic samples included in the second traffic sample set in each category is obtained, and the sample proportion distribution of the second traffic sample set under any of the feature dimensions is generated accordingly; The group stability index between the sample proportion distribution of the first traffic sample set in any characteristic dimension and the sample proportion distribution of the second traffic sample set in any characteristic dimension is determined as the characteristic distance between the first traffic sample set and the second traffic sample set in any characteristic dimension.

3. According to the method of claim 2, the classification strategy corresponding to any feature dimension includes: The traffic samples included in the traffic sample set to be classified are classified according to the preset categories included in any of the feature dimensions, and the classified categories are sorted in descending order according to the number of samples included, and all categories ranked below the preset ranking are merged into the same category.

4. The method according to claim 1, wherein generating the attenuation factor comprises: Determine a first attenuation factor and a second attenuation factor based on the number of samples of the first flow sample set and the number of samples of the second flow sample set, respectively, wherein the first attenuation factor is positively correlated to the number of samples of the first flow sample set, and the second attenuation factor is positively correlated to the number of samples of the second flow sample set; The attenuation factor is generated based on the first attenuation factor and the second attenuation factor.

5. The method according to claim 4, wherein determining the first attenuation factor and the second attenuation factor based on the number of samples of the first traffic sample set and the number of samples of the second traffic sample set respectively comprises: After the number of samples of the first traffic sample set is input into the attenuation factor generation function, the larger one of the first trend attention factor and the output value of the attenuation factor generation function is determined as the first attenuation factor; After the number of samples of the second traffic sample set is input into the attenuation factor generation function, the larger one of the second trend attention factor and the output value of the attenuation factor generation function is determined as the second attenuation factor; The generating the attenuation factor based on the first attenuation factor and the second attenuation factor comprises: The attenuation factor is determined as a product of the first attenuation factor and the second attenuation factor.

6. According to the method of claim 5, any one of the first trend attention factor and the second trend attention factor is zero or close to zero, the other one of the first trend attention factor and the second trend attention factor is greater than zero, and the ratio of the other trend attention factor to any one of the trend attention factors exceeds a preset value.

7. According to the method of claim 1, the server maintains a truncation threshold corresponding to the feature distance in at least two feature dimensions; the calculating the feature distance of the first traffic sample set and the second traffic sample set in at least one feature dimension, and determining the traffic anomaly index based on the feature distance in at least one feature dimension, comprises: Calculate at least two feature distances of the first traffic sample set and the second traffic sample set in the at least two feature dimensions respectively; In the case where any characteristic distance among the at least two characteristic distances exceeds the truncation threshold corresponding to the any characteristic distance, re-determining the value of the any characteristic distance as the truncation threshold corresponding to the any characteristic distance; The flow anomaly index is determined based on the re-determined at least two characteristic distances.

8. According to the method described in claim 7, the truncation threshold corresponding to the feature distance under any feature dimension maintained by the server is positively correlated with: the correlation between the one-dimensional traffic anomaly index of the historical traffic samples received by the server under any feature dimension and the attack of the historical network data automatic acquirer.

9. The method according to claim 7, wherein determining the flow anomaly index based on the at least two re-determined characteristic distances comprises: Determining a weighted sum of the at least two re-determined characteristic distances as the flow anomaly index; or, The quotient of the weighted sum of the re-determined at least two characteristic distances and the weighted sum of the truncation thresholds corresponding to the characteristic distances in the at least two characteristic dimensions is determined as the traffic anomaly index.

10. The method according to claim 1, wherein determining that there is a network data automatic acquirer attack in the first time period or the second time period comprises: When the number of samples in any one of the first traffic sample set and the second traffic sample set is greater than the number of samples in the other traffic sample set by a preset number, it is determined that a network data automatic acquirer attack exists on the server within a time period when the server receives any one of the traffic sample sets.

11. The method according to claim 1, further comprising: Before calculating the characteristic distances of the first traffic sample set and the second traffic sample set respectively under the at least one characteristic dimension, the identified network data automatic acquirer traffic samples included in the first traffic sample set and the second traffic sample set are deleted.

12. According to the method of claim 1, the first time period and the second time period are the same time period, the traffic samples included in the first traffic sample set are in either an anonymous state or a logged-in state, and the traffic samples included in the second traffic sample set are in the other of the anonymous state and the logged-in state; or, The first time period and the second time period are adjacent time periods, and the traffic samples included in the first traffic sample set and the second traffic sample set are both in an anonymous state or a logged-in state.

13. According to the method described in claim 1, the at least one feature dimension includes at least one of the following: the visitor's operating system, the visitor's operating system version, the visitor's browser, the visitor's browser version, the visitor's client version, the visitor's GPS location, the visitor's IP map location, the traffic access duration, and the cookie name.

14. A network data automatic acquisition device attack detection device, applied to a server, the server maintaining an alarm threshold, the device comprising: A traffic sample set acquisition unit, used to acquire a first traffic sample set received by the server in a first time period and a second traffic sample set received in a second time period; A flow anomaly index determination unit, configured to calculate a characteristic distance between the first flow sample set and the second flow sample set in at least one characteristic dimension, and determine a flow anomaly index based on the characteristic distance in the at least one characteristic dimension; The method of calculating the characteristic distance between the first traffic sample set and the second traffic sample set in at least one characteristic dimension respectively comprises: determining the characteristic distance between the first traffic sample set and the second traffic sample set in any characteristic dimension by a group stability index between the distribution characteristics of the first traffic sample set in any characteristic dimension and the distribution characteristics of the second traffic sample set in any characteristic dimension; the group stability index is obtained by the following calculation method: calculating the difference between the proportions of the first traffic sample set and the second traffic sample set in each same category in the same characteristic dimension, and summing up all the obtained difference values; A flow anomaly index adjustment unit, used to generate an attenuation factor and adjust the flow anomaly index according to the attenuation factor, wherein the attenuation factor is positively correlated to the number of samples in the first flow sample set and / or the number of samples in the second flow sample set; The network data automatic acquirer attack determination unit is used to determine that a network data automatic acquirer attack exists in the first time period or the second time period when the adjusted traffic anomaly index exceeds the alarm threshold.

15. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor implements the method according to any one of claims 1 to 13 by running the executable instructions.

16. A computer-readable storage medium having computer instructions stored thereon, characterized in that: When the instruction is executed by a processor, the steps of the method according to any one of claims 1 to 13 are implemented.

Images