Methods, devices, equipment and storage media for processing abnormal traffic data

Abstract

This disclosure provides a method, apparatus, device, and storage medium for processing abnormal traffic data, relating to the field of computer technology, and particularly to the field of internet security technology. The specific implementation scheme is as follows: acquiring the User Agent (UA) information of the traffic data; performing anomaly detection on the UA information to obtain the detection result; determining that the UA information is an abnormal UA based on the detection result, and identifying the abnormal traffic data missed by the anti-crawler strategy if the traffic data is identified as normal traffic data based on the anti-crawler strategy. In this disclosure, by detecting abnormal UA information, traffic data with abnormal UA information but judged as normal by the anti-crawler strategy can be filtered out. Therefore, the ability of the anti-crawler strategy to identify abnormal traffic data can be verified based on the abnormal UA, providing important data support for optimizing the anti-crawler strategy.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and more particularly to the field of internet security technology. Background Technology

[0002] With the development of internet technology, the amount of information on the internet is increasing, including much important and confidential information. Web crawlers can obtain and leak this important information, causing harm to content providers and users. Summary of the Invention

[0003] This disclosure provides a method, apparatus, device, and storage medium for processing abnormal traffic data.

[0004] According to one aspect of this disclosure, a method for processing abnormal traffic data is provided, comprising:

[0005] Obtain UA (User Agent) information for traffic data;

[0006] Anomaly detection is performed on the UA information, and the detection results are obtained.

[0007] If the detection results indicate that the UA information is an abnormal UA, and the traffic data is identified as normal traffic data based on the anti-crawler strategy, then it is determined that the anti-crawler strategy missed detecting the abnormal traffic data.

[0008] According to another aspect of this disclosure, an apparatus for processing abnormal traffic data is provided, comprising:

[0009] The acquisition module is used to obtain the user agent (UA) information of traffic data;

[0010] The UA information detection module is used to detect anomalies in UA information and obtain the detection results.

[0011] The missed detection determination module is used to determine the abnormal traffic data missed by the anti-crawler strategy when the UA information is determined to be an abnormal UA based on the detection results and the traffic data is identified as normal traffic data based on the anti-crawler strategy.

[0012] According to another aspect of this disclosure, an electronic device is provided, comprising:

[0013] At least one processor; and

[0014] The memory is communicatively connected to the at least one processor; wherein,

[0015] The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the abnormal traffic data processing method of the present disclosure embodiments.

[0016] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause the computer to perform a method for processing abnormal traffic data according to an embodiment of this disclosure.

[0017] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements a method for processing abnormal traffic data according to embodiments of this disclosure.

[0018] In this disclosure, when using common anti-crawler strategies for traffic anomaly detection, some false negatives may occur. This disclosure provides a method for verifying anti-crawler strategies by detecting abnormal User Agents (UAs) from the UA information dimension.

[0019] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0020] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0021] Figure 1 This is a flowchart illustrating a method for processing abnormal traffic data according to an embodiment of the present disclosure;

[0022] Figure 2 This is a flowchart illustrating the anomaly detection of UA information according to another embodiment of the present disclosure;

[0023] Figure 3 This is a flowchart illustrating the anomaly detection of UA information according to another embodiment of the present disclosure;

[0024] Figure 4 This is a schematic diagram of the process for detecting anomalies in UA information according to another embodiment of this disclosure;

[0025] Figure 5 This is a flowchart illustrating the detection of UA information using various detection methods according to another embodiment of this disclosure;

[0026] Figure 6 This is a structural diagram of an apparatus for processing abnormal traffic data according to another embodiment of the present disclosure;

[0027] Figure 7 This is a block diagram of an electronic device used to implement the abnormal traffic data processing method of the embodiments of this disclosure. Detailed Implementation

[0028] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0029] With the development of internet technology, content providers mostly use web pages to deliver content to users. To protect the rights of content providers, it is necessary to monitor access traffic data. This is to prevent malicious traffic from consuming server processing resources and to prevent core content assets from being scraped.

[0030] Web crawlers are a method of generating malicious traffic. A web crawler is a program that automatically extracts data from web pages, collecting and organizing the data, which can greatly save on labor costs. However, many malicious users use web crawlers to obtain important information such as user information, company business data, and intellectual property data, resulting in the leakage of user information and damage to company interests.

[0031] Therefore, anti-crawler strategies are needed to detect and identify web crawlers to prevent them from obtaining the aforementioned important information. Some characteristics of web crawler traffic data differ from those of real users, such as IP (Internet Protocol) addresses. Since web crawlers open a large number of web pages in a short period to collect data, the same IP address making multiple requests to the server within a short time can be identified as abnormal traffic data from a web crawler. As crawler strategies may be constantly updated, anti-crawler strategies also need continuous adjustment and optimization. There are many anti-crawler strategies available; verifying the effectiveness of each strategy can better assist in its optimization.

[0032] User Agent (UA) information is a special string header that allows the server to identify the device information and browsing information of the client. In cases of abnormal UA information, the corresponding traffic data is also abnormal. This disclosure provides a scheme for verifying anti-crawler strategies based on UA ​​information, which can provide direct evidence of abnormal traffic and help verify and optimize anti-crawler strategies.

[0033] Therefore, in order to verify and improve the ability to identify anti-scraping strategies, such as Figure 1 As shown, the present disclosure provides a method for processing abnormal traffic data using UA information. This method includes:

[0034] S101. Obtain UA information for traffic data.

[0035] S102. Perform anomaly detection on the UA information and obtain the detection results.

[0036] S103. If the UA information is determined to be an abnormal UA based on the detection results, and the traffic data is identified as normal traffic data based on the anti-crawler strategy, then it is determined that the anti-crawler strategy missed detecting the abnormal traffic data.

[0037] Once the traffic data is determined to be abnormal, it can be reported, and staff will take action, such as blocking the same IP address from accessing the webpage.

[0038] In this embodiment, abnormal user agents (UAs) are detected through UA information, allowing for relatively accurate identification of abnormal traffic data. Further verification of anti-crawler strategies using these abnormal UAs allows for the filtering out of traffic data with abnormal UAs that the anti-crawler strategy deems normal. This verifies the anti-crawler strategy's ability to identify abnormal traffic data. UA anomalies serve as direct evidence of missed detections by the anti-crawler strategy, helping to identify areas for optimization and improve its ability to identify abnormal traffic data.

[0039] To further improve the ability to detect anomalies in UA information, in some embodiments, when detecting UA information, anomaly detection is performed based on at least one of the following first detection method, second detection method, and third detection method to obtain the detection result:

[0040] in,

[0041] In the first detection method, anomaly detection is performed on the UA information based on the target string representing the number of targets in the UA information;

[0042] In the second detection method, anomalies in UA information are detected based on anomaly UA feature sets.

[0043] The third detection method uses a neural network model to detect anomalies in UA information.

[0044] In this embodiment, a single detection method can easily and conveniently identify abnormal User Agents (UAs) to facilitate the verification of anti-crawler strategies. Alternatively, multiple detection methods can be used in combination to detect abnormal UA information, thereby improving the accuracy of abnormal UA detection and thus increasing the efficiency of anti-crawler strategy verification. Furthermore, using multiple detection methods in combination leverages the advantages of each method, maximizing the detection of abnormal UAs and improving the detection efficiency for different types of abnormal UAs. This makes the method of this embodiment more widely applicable, facilitating the effective verification of anti-crawler strategies.

[0045] In some embodiments, when the UA information is not empty, the above-described set of detection methods is used to detect abnormal UAs. To further improve detection efficiency, before using the set of detection methods, it is determined whether the UA information is empty. If it is not empty, the set of detection methods can be used to further detect anomalies in the UA information.

[0046] The various detection methods will be introduced below.

[0047] 1) First detection method

[0048] In the first detection method, anomaly detection is performed on the UA information by detecting the target string of the UA information. The target string is the shortest string that can achieve anomaly detection of the UA information. The length of this shortest string can be determined empirically; in one possible implementation, the first 12 characters of the UA information can be selected as the target string.

[0049] In other implementations, cluster analysis can be performed on the UA information of abnormal UAs, filtering out common strings in each category and counting the number of characters in the common strings. Then, the longest character count among the common strings is selected as the character count of the target string (i.e., the target number). Alternatively, the average character count of the common strings can be selected as the character count of the target string.

[0050] In this embodiment of the disclosure, a blacklist can be used to detect anomalies in target strings within the User Agent (UA) information. The blacklist records the number of target strings containing abnormal UA information; this can be understood as storing target strings containing known abnormal UA information in the blacklist.

[0051] Therefore, the blacklist records only a portion of the abnormal UA information. Compared to recording the entire UA information, this not only reduces the amount of data in the blacklist and saves storage resources, but also allows for the detection of the same type of abnormal UA through the same record in the blacklist, thus improving detection efficiency.

[0052] Specifically, anomaly detection of UA information based on the target string representing the number of targets can be implemented as follows: Figure 2 As shown, it includes:

[0053] S201. Extract the target quantity string from the UA information to obtain the target string.

[0054] For example, starting from the first character of the UA information, extract n characters to obtain the target string, where n is the target quantity.

[0055] S202. Perform a matching operation between the target string and the blacklist.

[0056] The matching method can be selected as character-by-character matching, which means matching the target string against each record in the blacklist. If no record matching the target string is found after matching all records in the blacklist, it is determined that the target string is not matched in the blacklist. If any record is matched, it is determined that the target string is matched in the blacklist, and the matching operation can be terminated.

[0057] In other implementations, features can be extracted from the string to obtain the feature vector of the target string, and the similarity between the target string and the feature vector of each record in the blacklist can be calculated. If the similarity between any record and the target string is greater than a set threshold, then the target string is determined to be matched in the blacklist.

[0058] S203. If the target string is matched in the blacklist, determine that the UA information is an abnormal UA.

[0059] Compared to matching the complete User Agent (UA) information against a blacklist, matching a subset of the UA information—the target string—against the blacklist reduces the amount of information to be matched, thus improving matching efficiency. By matching against the blacklist, abnormal UAs belonging to the target string category can be identified more accurately, thereby accurately detecting missed abnormal traffic data and providing support for verifying anti-scraping strategies.

[0060] Abnormal User Agents (UAs) may change constantly, and a static blacklist cannot adapt to such changes. To ensure the timeliness of the blacklist and make the currently used blacklist applicable to the current situation, the blacklist is updated periodically in this embodiment of the disclosure.

[0061] During implementation, the blacklist for the next statistical period can be updated based on the traffic data or visited pages generated within the current statistical period. This blacklist is used by the primary detection method to detect abnormal User Agents (UAs). In other words, the blacklist can be updated each statistical period based on the abnormal traffic situation within that period, so that the new blacklist can be used to detect abnormal UAs in the next statistical period, thereby ensuring the timeliness of the blacklist.

[0062] The statistical period can be determined according to the actual situation. For example, if the number of visitors to a website or application is large, the statistical period can be set to 1 hour or even less; if the number of visitors to a website or application is small, the statistical period can be set to 3 hours or a longer period of time, thereby reducing the use of server resources.

[0063] Periodically analyze traffic data and accessed pages, and update the blacklist accordingly. This allows the blacklist to be updated based on current conditions, ensuring it adapts to changing abnormal user agents (UAs), thereby improving the efficiency and accuracy of anomaly detection based on the blacklist. Furthermore, it provides a solid data foundation for validating the effectiveness of anti-crawler strategies.

[0064] In summary, the exemplary method of this disclosure can update the blacklist based on traffic data within a statistical period or the pages accessed within the statistical period. Based on this, this disclosure can exemplarily provide two methods for updating the blacklist required for the next statistical period.

[0065] Method 1 for updating the blacklist: Update the blacklist based on traffic data generated within the current statistical period.

[0066] For example, a blacklist can be determined by the proportion of abnormal traffic in traffic data, which can be implemented as follows:

[0067] Step A1: Obtain traffic data containing the target string within the current statistical period to obtain the traffic dataset to be processed.

[0068] For example, if m traffic data points are generated within a statistical period, and m User Agent (UA) information points are extracted from them, with one string extracted from each UA information point, then traffic data points with the same string can be counted. Based on the target string, the corresponding traffic dataset to be processed can be obtained.

[0069] Step A2: Determine the proportion of abnormal traffic in the traffic dataset to be processed.

[0070] The abnormal traffic in the traffic dataset to be processed is not limited to abnormal traffic identified by the first detection method, but can also be abnormal traffic identified by the second or third detection method, or even abnormal traffic identified by other abnormal traffic detection methods. These other abnormal traffic detection methods include, for example, abnormal traffic detected using anti-crawler strategies, such as detecting IP addresses to identify the same IP address making multiple requests to the server within a short period as abnormal traffic.

[0071] Step A3: If the percentage of abnormal traffic exceeds the percentage threshold and the target string is not in the blacklist, update the target string to the blacklist required for the next statistical period.

[0072] In some embodiments, the percentage threshold can be determined based on the actual situation of the website or application. For example, for websites or applications containing relatively little important information, the percentage threshold can be set larger; while for websites or applications containing a lot of important information, or where the leakage of important information would cause huge losses, the percentage threshold can be set smaller. Different percentage thresholds determine the number of records in the blacklist. The smaller the percentage threshold, the larger the number of records in the blacklist, the more lenient the judgment of abnormal User Agents (UAs), the more traffic data will be judged as abnormal UAs, and more abnormal UAs can be detected. The larger the percentage threshold, the fewer the number of records in the blacklist, and the less traffic data will be judged as abnormal UAs. In implementation, a reasonable percentage threshold can be set according to actual needs.

[0073] The method provided in this disclosure for updating the blacklist ensures that newly generated abnormal User Agents (UAs) are promptly added to the blacklist for the next statistical period, guaranteeing the blacklist's timeliness. This timely update allows the blacklist to more accurately identify abnormal traffic while reducing the possibility of misjudging normal traffic data, thus providing a solid data foundation for verifying anti-crawler strategies. Furthermore, by setting a percentage threshold, the method can be flexibly applied to different applications or websites.

[0074] Method 2 for updating the blacklist: Update the blacklist based on the pages accessed within the statistical period.

[0075] This method can update the blacklist by counting the number of visited pages containing the target string, and can be implemented as follows:

[0076] Step B1: Obtain the User Agent (UA) information from the traffic data of the pages accessed within the current statistical period.

[0077] Step B2: Determine the number of accessed pages whose User Agent (UA) information contains the target string.

[0078] Step B3: If the number of accessed pages containing the target string exceeds the page count threshold and the target string is not in the blacklist, add the target string to the blacklist required for the next statistical period.

[0079] Because web crawlers access far more pages per unit of time than human users—for example, a web crawler written in Java can access tens of thousands of web pages per second—abnormal data can be identified by counting the number of pages accessed.

[0080] In this embodiment of the disclosure, by counting the number of page visits for the same target string, the target string whose page visit count exceeds the page count threshold can be added to the blacklist.

[0081] In summary, page view count is an effective means of identifying abnormal traffic. By counting the number of pages accessed within a statistical period, new abnormal User Agents (UAs) can be identified and updated to the blacklist, thereby improving the timeliness of the blacklist and the efficiency of detecting abnormal UAs. This provides better direct evidence for abnormal traffic and helps verify anti-crawler strategies.

[0082] It is understandable that the aforementioned Implementation Method 1 and Implementation Method 2 update the blacklist in two different ways. In implementation, both Implementation Method 1 and Implementation Method 2 can be used simultaneously to further improve the timeliness of the blacklist.

[0083] It should be noted that in this embodiment of the disclosure, the blacklist of the next statistical period can be inherited from the blacklist of the previous statistical period. For example, if the blacklist of the current statistical period includes abnormal UA1 and abnormal UA2, and abnormal UA3 is added in the current statistical period, then the blacklist of the next statistical period will include abnormal UA1, abnormal UA2, and abnormal UA3.

[0084] As the number of abnormal User-Agents (UAs) increases, the blacklist grows longer, and some abnormal UAs may be rarely used. Therefore, the blacklist can be used to prioritize matching abnormal UAs. For example, the frequency of abnormal UAs matched within a specified time period can be statistically analyzed, and the matching priority of abnormal UAs recorded in the blacklist can be set. Understandably, abnormal UAs with higher matching frequencies have higher matching priority. This allows for matching against frequently used abnormal UAs in the blacklist, quickly determining whether UA information is abnormal, reducing the number of matches between the target string and each record in the blacklist, and improving the efficiency of blacklist usage.

[0085] In addition to matching frequency, newly added abnormal User Agents (UAs) should also have a higher matching priority, as they may represent recently developed and frequently used methods by malicious users. During implementation, the matching priority can be determined based on the time the UA was added to the blacklist; the later the UA was added, the higher the matching priority.

[0086] Therefore, this embodiment of the disclosure provides two exemplary indicators for determining matching priority: matching frequency and the time of addition to the blacklist. In implementation, other indicators can be added according to actual needs. With multiple indicators, the matching priority of each record (i.e., each abnormal User Agent) in the blacklist can be determined based on each indicator. Each indicator has a preset weight. For each record in the blacklist, the matching priorities obtained from each indicator can be weighted and summed based on the weights to determine the final matching priority of that record.

[0087] The preceding text described the first detection method, which uses a blacklist to detect anomalies in the target string within the User Agent (UA) information. To improve the efficiency of this first detection method, this embodiment also provides a whitelist. The target string is also matched against the whitelist; if the target string matches the whitelist, the UA information is determined to be a normal UA.

[0088] In practice, records in the whitelist cannot be directly identified by the proportion of abnormal traffic or the number of page views, unlike the blacklist. However, to improve the efficiency of determining or updating the whitelist, this embodiment can extract target strings with a low proportion of abnormal traffic (e.g., below a first threshold) and a low number of page views (e.g., below the second number of pages) over a long period and add them to the whitelist. Furthermore, to ensure the accuracy of whitelist members, the extracted target strings and their related data can be collected so that staff can manually determine whether to add them to the whitelist based on expert knowledge. Simultaneously, to facilitate staff in analyzing website or application data, the target strings corresponding to the web crawlers used by staff can be added to the whitelist. This avoids misidentifying web crawlers used by staff as abnormal traffic, reducing the use of server resources.

[0089] In summary, by setting up a whitelist, normal traffic can be filtered out, improving the efficiency of detecting anomalies in User Agent (UA) information.

[0090] In addition, using a whitelist can also prevent normal traffic or web crawlers used by websites or applications from being mistakenly identified as abnormal traffic.

[0091] When using the first detection method, you can first use a whitelist to detect the target string. If the target string is not included in the whitelist, then use a blacklist for anomaly detection.

[0092] Of course, both blacklists and whitelists can be used simultaneously for anomaly detection on the target string. Alternatively, the blacklist can be used first for anomaly detection, and if it is determined that the target string is not included in the blacklist, then the whitelist can be used for detection.

[0093] In summary, the first detection method can provide three detection results based on blacklists and whitelists: first, the UA information is determined to be an abnormal UA; second, the UA information is determined to be a normal UA; and third, unknown, meaning the target string is neither included in the blacklist nor the whitelist. Therefore, other detection methods can be used for joint detection of UA information. The second detection method is explained below.

[0094] 2) Second detection method

[0095] The second detection method uses anomaly UA feature sets to detect anomalies in UA information. In practice, by statistically analyzing a large number of abnormal traffic UAs, UAs that meet certain syntactic forms can be identified as abnormal UAs. For example, UAs with zero uppercase characters are abnormal UAs; similarly, UAs with zero lowercase characters are abnormal UAs. Therefore, the features of abnormal UAs can be extracted to form an abnormal UA feature set. When UA information meets any one of these features, it can be identified as an abnormal UA, such as... Figure 3 As shown, the method can be implemented as follows:

[0096] S301. Compare the UA information with at least one syntactic feature from the abnormal UA feature set.

[0097] S302. If the UA information matches any syntactic feature, determine that the UA information is an abnormal UA.

[0098] In summary, comparing the syntactic features of abnormal User Agents (UAs) with UA information can identify abnormal UAs, improving the efficiency of abnormal UA detection. Since the abnormal UA feature set contains multiple abnormal UA features, comparing them separately with UA information can detect various types of abnormal UAs, improving the accuracy of abnormal UA judgment. Furthermore, combining it with other detection methods can better detect different abnormal UAs, thus laying the foundation for accurately and effectively identifying missed abnormal traffic and verifying the effectiveness of anti-crawler strategies.

[0099] 3) Third detection method

[0100] The third detection method uses a neural network model to detect anomalies in UA information. This neural network model can be trained using the following methods:

[0101] Step C1: Obtain training samples. The training samples include UA information and the corresponding training labels. The training labels are used to indicate whether the UA information is an abnormal UA.

[0102] Step C2: Input the training sample into the neural network model to be trained, and obtain the classification result of the UA information in the training sample by the neural network model to be trained. The neural network model to be trained is used to classify the input UA information as abnormal UA or normal UA.

[0103] Step D3: Determine the loss value based on the classification results and training labels.

[0104] Step D4: Adjust the model parameters of the neural network model to be trained based on the loss value. If the training convergence condition is met, end the training to obtain a neural network model that can detect anomalies in UA information.

[0105] In other words, the training samples in the training sample set include both normal UAs and abnormal UAs. The neural network model to be trained can predict whether each training sample in the training sample set is a normal UA or an abnormal UA. When the prediction result matches the label, i.e., the prediction is correct, a reward is given through the loss function. When the prediction result does not match the label, i.e., the prediction is incorrect, a penalty is given through the loss function. This allows the model parameters of the neural network model to be trained to be adjusted, enabling the model to learn the ability to detect abnormal UAs.

[0106] Assuming the neural network model is well-trained, Figure 4 As shown, this neural network model can be used to detect abnormal user agents (UAs), including:

[0107] S401. Input the UA information into the neural network model to obtain the classification result of the UA information output by the neural network model.

[0108] S402. If the classification result indicates that the UA information is an abnormal UA, then the UA information is determined to be an abnormal UA; if the classification result indicates that the UA information is a normal UA, then the UA information is determined to be a normal UA.

[0109] Since the User-Agent (UA) information consists of a string with a specific meaning, the neural network model selected in this embodiment is an RNN (Recurrent Neural Network). The RNN model can understand the UA information based on the context of the characters, thus improving the efficiency of UA anomaly detection.

[0110] In this embodiment of the disclosure, the detection of abnormal User Agents (UAs) using a neural network model can significantly save server resources and manpower costs, improve the detection efficiency of abnormal UAs, and, in combination with other detection methods, effectively detect abnormal UAs and improve the identification efficiency of abnormal traffic.

[0111] The above describes exemplary implementations of various detection methods for anomaly detection of User Agents (UAs). After obtaining the detection results of multiple detection methods on the UA information, if the results obtained by these methods are the same, then that result is used as the final detection result to determine whether the UA information is an abnormal UA. If the results obtained by these detection methods are different, the detection results of various methods can be summarized and stored in a summary set for analysis by professionals.

[0112] In another embodiment, the final detection result can be determined based on the weight of each detection method, and it can be determined whether the traffic data is abnormal traffic. The weights can be determined based on the actual situation of the website or application, or they can be determined according to a weight determination model.

[0113] For example, the confidence level of the detection result from the neural network model is a0, and different confidence levels correspond to different scores, let's say a score of a1, with a weight of b1. The detection result from the first detection method also has a corresponding score, for example, a score of a2, with a corresponding weight of b2. The detection result from the second detection method also has a corresponding score, for example, a score of a3, with a corresponding weight of b3. The final detection score can then be obtained by weighted summation. If the detection score is greater than a specified threshold, the UA information can be determined to be an abnormal UA; if the detection score is less than or equal to the specified threshold, the UA information can be determined to be a normal UA.

[0114] In other embodiments, to reduce processing resource consumption and improve detection efficiency, different detection methods can be configured in a specific order. This detection order is then used to sequentially detect anomalies in the UA information. For example, the UA information is detected by executing the first and second detection methods sequentially, and the first and third detection methods are executed in parallel. Figure 5 As shown:

[0115] After receiving the reported UA information, first execute S501 to check if the UA information is empty.

[0116] If the UA information is empty, it is directly identified as an abnormal UA. If the UA information is not empty, the first detection method is used to detect it first, and S506 is executed to perform anomaly detection on the UA information using a neural network model.

[0117] When using the first detection method, refer to Figure 5 Execute S502 to extract the target string.

[0118] S503. Check if the target string matches the blacklist. If it matches, confirm it as an abnormal User Agent (UA). If it does not match the blacklist, match it against the whitelist. If it does not match the whitelist, use the second detection method to detect it.

[0119] When using the second detection method, refer to Figure 5 The process can be executed as follows: In S504, the UA information is compared with the abnormal UA feature set. If the UA information matches any syntactic feature, it is confirmed as an abnormal UA. If it does not match, it is determined to be a normal UA.

[0120] In S505, the results of various detection methods are summarized and analyzed. For normal User Agents (UAs), refer to... Figure 5 For abnormal User Agents (UAs), execute S507 to report a risk-free status; for abnormal UAs, execute S508 to report an abnormal status.

[0121] Furthermore, in this embodiment of the disclosure, in order to understand the degree of risk, a risk score can be determined for the detection result and output. For example, if the result matches a blacklist, a score of c1 is output, and if the result matches a whitelist, a score of c2 is output, where c2 is less than c1.

[0122] For each anomalous UA feature set, there is a corresponding risk score. When any anomalous UA feature is matched, the risk score output by that anomalous UA is the risk score corresponding to that anomalous UA feature.

[0123] For neural network models, risk scores can be set for identifying abnormal user agents (UAs) and for identifying normal UAs. Of course, neural network models may also not have corresponding risk scores.

[0124] In some implementations, risk scoring can be used to further determine whether the UA information is ultimately abnormal. For example, a threshold is determined based on the lowest risk score corresponding to the blacklist and abnormal UA features. After obtaining the risk score, if it is greater than or equal to the threshold, it is judged as an abnormal UA; if it is less than the threshold, it is a normal UA. For example, a blacklist hit score is 3, a whitelist hit score is 1, and hit scores for different abnormal UA features range from 3 to 7. The threshold can then be set to 2; if the score is greater than 2, the UA information is an abnormal UA; if it is less than or equal to 2, the UA information is a normal UA.

[0125] In some embodiments, after anomaly detection of User Agent (UA) information, abnormal UAs can be statistically summarized to provide important clues for risk tracing. For example, abnormal UAs can be statistically analyzed based on dimensions such as IP address and user account to provide clues for risk tracing. Specifically, this can be implemented as follows:

[0126] Step E1: Obtain the unique identifier of the source of the traffic data;

[0127] The unique identifier is used to confirm the source of traffic. It can be information such as IP address or user account mentioned above, or it can be information that can uniquely identify the source of data, such as MAC address (Media Access Control).

[0128] Step E2: Based on the unique identifier, calculate the abnormal UA carry-over rate corresponding to the unique identifier within a specified time period, as reference information for risk tracing.

[0129] For example, for pages accessed from the same source, the number of page views (PVs) that generate abnormal user agents (UAs) can be counted on a daily basis, and the total number of PVs from that source within that daily period can be counted. The abnormal user agent carryover rate can be obtained by dividing the number of PVs with abnormal UAs by the total number of PVs.

[0130] During implementation, the abnormal UA portability rate can be statistically analyzed from two dimensions: IP address and account.

[0131] In this embodiment of the disclosure, the abnormal UA carry-over rate of the data source is determined, providing direct and reliable reference evidence for risk tracing; this enables staff to more accurately judge the risk status of the data source, so as to determine the direction of anti-crawler strategy adjustment and improve the optimization efficiency of the anti-crawler strategy.

[0132] During implementation, a unique identifier with a high abnormal user agent (UA) carry rate can be pushed to the user so that the user can execute the corresponding access policy and protect core assets.

[0133] In some embodiments, the risk level can also be determined based on the abnormal UA carryover rate. Multiple risk levels can be set. Taking low-risk, medium-risk, and high-risk levels as an example, determining the risk level of a unique identifier can be implemented as follows: if the abnormal UA carryover rate of the unique identifier is lower than a first risk threshold, it is determined to be low-risk; if the abnormal UA carryover rate of the unique identifier is higher than the first risk threshold but lower than a second risk threshold, it is determined to be medium-risk; and if the abnormal UA carryover rate of the unique identifier is higher than the second risk threshold, it is determined to be high-risk. After obtaining the risk level of the unique identifier, it is stored in the corresponding database for subsequent use.

[0134] For example, a unique identifier rated as low risk indicates that the unique identifier is relatively safe; a unique identifier rated as medium risk indicates that the unique identifier needs attention; and a unique identifier rated as high risk can be reported directly so that staff can focus on this type of anti-scraping strategy and adjust and optimize it.

[0135] Of course, in other embodiments, the abnormal UA carry-over rate can also be used as a measure of risk level. This provides staff with a unique identifier to monitor for high abnormal UA carry-over rates and offers relevant data to help optimize and adjust anti-crawler strategies.

[0136] To efficiently optimize anti-scraping strategies, after identifying missed anomalies, priority anti-scraping strategies that need optimization can be provided to staff. Specifically, the following can be implemented:

[0137] Method F1: Statistically assess the degree of missed detection by anti-crawler strategies.

[0138] In implementation, for anti-crawler strategies, the number of missed abnormal traffic data within a set time period can be aggregated, or the page views (PV) of missed abnormal traffic data can be counted as the degree of missed detection. For anti-crawler strategies, the aggregation granularity can also be at the IP level or the UID (User Identification) level. This yields the degree of missed detection at the IP level or UID level.

[0139] Method F2: Determine the priority of anti-crawler strategies to be optimized based on the degree of missed detections.

[0140] During implementation, the higher the rate of missed detections, the higher the priority for optimizing the anti-scraping strategy. Optimization can involve adjusting the existing anti-scraping strategy or developing a new one.

[0141] For example, given the IP-level miss rate, sort the anti-crawler strategies in descending order of miss rate to provide staff with the top-ranked strategies, so that staff can understand which anti-crawler strategies need to be optimized first.

[0142] By statistically analyzing the failure rate of anti-scraping strategies and determining optimization priorities based on this rate, we can clarify the optimization direction for staff and improve optimization efficiency.

[0143] Based on the same technical concept, this disclosure also provides an apparatus for processing abnormal traffic data, such as... Figure 6 As shown, the device includes:

[0144] Module 601 is used to obtain the User Agent (UA) information of traffic data;

[0145] The UA information detection module 602 is used to detect anomalies in UA information and obtain detection results.

[0146] The missed detection determination module 603 is used to determine the abnormal traffic data missed by the anti-crawler strategy when the UA information is determined to be an abnormal UA based on the detection results and the traffic data is identified as normal traffic data based on the anti-crawler strategy.

[0147] In some embodiments, the UA information detection module 602 is used for:

[0148] Anomaly detection of UA information is performed based on at least one of the following first, second, and third detection methods, and the detection results are obtained:

[0149] in,

[0150] In the first detection method, anomaly detection is performed on the UA information based on the target string representing the number of targets in the UA information;

[0151] In the second detection method, anomalies in UA information are detected based on anomaly UA feature sets.

[0152] The third detection method uses a neural network model to detect anomalies in UA information.

[0153] In some embodiments, the device further includes:

[0154] The update module is used to update the blacklist required for the next statistical period based on the traffic data or pages accessed during the current statistical period. The blacklist is used by the first detection method to detect abnormal user agents.

[0155] In some embodiments, the UA information detection module 602 is used for:

[0156] Extract the target quantity string from the UA information to obtain the target string;

[0157] Perform a matching operation between the target string and the blacklist;

[0158] If the target string is matched in the blacklist, the UA information is determined to be an abnormal UA.

[0159] In some embodiments, the update module is configured to:

[0160] Obtain traffic data containing the target string within the current statistical period to obtain the traffic dataset to be processed;

[0161] Determine the percentage of abnormal traffic in the traffic dataset to be processed;

[0162] If the percentage of abnormal traffic exceeds the percentage threshold and the target string is not in the blacklist, the target string will be updated to the blacklist required for the next statistical period.

[0163] In some embodiments, the update module is configured to:

[0164] Retrieve User Agent (UA) information from the traffic data of pages accessed within the current statistical period;

[0165] Determine the number of accessed pages whose User Agent (UA) information contains the target string;

[0166] If the number of accessed pages containing the target string exceeds the page count threshold and the target string is not in the blacklist, the target string will be added to the blacklist required for the next statistical period.

[0167] In some embodiments, the UA information detection module 602 is used for:

[0168] The UA information is compared with at least one syntactic feature in the abnormal UA feature set; if the UA information matches any syntactic feature, the UA information is determined to be an abnormal UA.

[0169] In some embodiments, the UA information detection module 602 is used for:

[0170] Perform a matching operation between the target string and the whitelist;

[0171] If the target string matches the whitelist, the UA information is determined to be a normal UA.

[0172] In some embodiments, the device further includes:

[0173] The identifier acquisition module is used to obtain a unique identifier of the source of traffic data;

[0174] The first statistics module is used to calculate the abnormal user agent (UA) carry rate corresponding to a unique identifier within a specified time period based on the unique identifier, as reference information for risk tracing.

[0175] In some embodiments, the device further includes:

[0176] The second statistics module is used to calculate the degree of missed detection by anti-crawler strategies;

[0177] The optimization module is used to determine the priority of anti-scraping strategies that need to be optimized based on the degree of missed detections.

[0178] The specific functions and examples of each module and submodule of the apparatus in this disclosure can be found in the relevant descriptions of the corresponding steps in the above method embodiments, and will not be repeated here.

[0179] The acquisition, storage, and application of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0180] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0181] Figure 7 A schematic block diagram of an example electronic device 700 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0182] like Figure 7As shown, device 700 includes a computing unit 701, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 702 or a computer program loaded into random access memory (RAM) 703 from storage unit 708. RAM 703 may also store various programs and data required for the operation of device 700. The computing unit 701, ROM 702, and RAM 703 are interconnected via bus 704. Input / output (I / O) interface 705 is also connected to bus 704.

[0183] Multiple components in device 700 are connected to I / O interface 705, including: input unit 706, such as keyboard, mouse, etc.; output unit 707, such as various types of monitors, speakers, etc.; storage unit 708, such as disk, optical disk, etc.; and communication unit 709, such as network card, modem, wireless transceiver, etc. Communication unit 709 allows device 700 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0184] The computing unit 701 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 701 performs the various methods and processes described above, such as the method for processing abnormal traffic data. For example, in some embodiments, the method for processing abnormal traffic data may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 708. In some embodiments, part or all of the computer program may be loaded and / or installed on device 700 via ROM 702 and / or communication unit 709. When the computer program is loaded into RAM 703 and executed by the computing unit 701, one or more steps of the method for processing abnormal traffic data described above may be performed. Alternatively, in other embodiments, the computing unit 701 may be configured to perform the method for processing abnormal traffic data by any other suitable means (e.g., by means of firmware).

[0185] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0186] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0187] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0188] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0189] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0190] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0191] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0192] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A method for processing traffic data, comprising: User agent (UA) information for obtaining traffic data; Anomaly detection is performed on the UA information to obtain the detection results; If the UA information is determined to be an abnormal UA based on the detection results, and the traffic data is identified as normal traffic data based on the anti-crawler strategy, then the anti-crawler strategy is determined to have missed detecting abnormal traffic data. The abnormal traffic data is used to verify and optimize the anti-crawler strategy. The detection methods include: A first detection method for detecting anomalies in the UA information based on the target string containing the target quantity; A second detection method for detecting anomalies in the UA information based on an abnormal UA feature set; A third detection method based on a neural network model to detect anomalies in the UA information; The first detection method and the third detection method are executed in parallel, and the first detection method is executed before the second detection method. It also includes: based on the traffic data generated or the pages accessed in the current statistical period, determining the abnormal UA described in the current statistical period, updating the abnormal UA to the blacklist required for the next statistical period, and the blacklist is used by the first detection method to detect abnormal UA. The matching priority of each abnormal user agent in the blacklist is determined based on the matching frequency and / or the time when it was added to the blacklist.

2. The method according to claim 1, wherein, Anomaly detection is performed on the UA information based on the target string representing the number of targets, including: The target string is obtained by extracting the target quantity string from the UA information; The target string is matched against the blacklist. If the target string is matched in the blacklist, the UA information is determined to be an abnormal UA.

3. The method according to claim 1, wherein, Based on the traffic data generated within the current statistical period, determine the blacklist required for the next statistical period, including: Obtain traffic data containing the target string within the current statistical period to obtain the traffic dataset to be processed; Determine the proportion of abnormal traffic in the traffic dataset to be processed; If the proportion of abnormal traffic exceeds the proportion threshold and the target string is not in the blacklist, the target string will be updated to the blacklist required for the next statistical period.

4. The method according to claim 1, wherein, Based on the pages accessed during the current statistical period, determine the blacklist required for the next statistical period, including: Retrieve User Agent (UA) information from the traffic data of pages accessed within the current statistical period; Determine the number of accessed pages whose User Agent (UA) information contains the target string; If the number of accessed pages containing the target string exceeds the page count threshold and the target string is not in the blacklist, the target string will be added to the blacklist required for the next statistical period.

5. The method according to claim 1, wherein, Anomaly detection of the UA information based on anomaly UA feature sets includes: The UA information is compared with at least one syntactic feature in the abnormal UA feature set; If the UA information matches any syntactic feature, the UA information is determined to be an abnormal UA.

6. The method according to claim 1, wherein anomaly detection is performed on the UA information based on the target string representing the number of targets in the UA information, comprising: The target string is matched against the whitelist. If the target string matches the whitelist, the UA information is determined to be a normal UA.

7. The method according to any one of claims 1-6, further comprising: Obtain the unique identifier of the source of the traffic data; Based on the unique identifier, the abnormal UA carrying rate corresponding to the unique identifier within a specified time period is calculated as reference information for risk tracing.

8. The method according to any one of claims 1-6, further comprising: The degree of missed detection by the aforementioned anti-crawler strategy was statistically analyzed; The priority of optimizing the anti-crawler strategy is determined based on the degree of missed detections.

9. A device for processing traffic data, comprising: The acquisition module is used to obtain the user agent (UA) information of traffic data; The UA information detection module is used to detect anomalies in the UA information and obtain detection results. The missed detection determination module is used to determine that the UA information is an abnormal UA based on the detection result, and that the traffic data is identified as normal traffic data based on the anti-crawler strategy, and to determine that the anti-crawler strategy has missed the abnormal traffic data. The abnormal traffic data is used to verify and optimize the anti-crawler strategy. The detection methods include: A first detection method for detecting anomalies in the UA information based on the target string containing the target quantity; A second detection method for detecting anomalies in the UA information based on an abnormal UA feature set; A third detection method based on a neural network model to detect anomalies in the UA information; The first detection method and the third detection method are executed in parallel, and the first detection method is executed before the second detection method. Also includes: The update module is used to determine the abnormal User Agent (UA) in the current statistical period based on the traffic data or pages accessed in the current statistical period, and update the abnormal UA to the blacklist required for the next statistical period. The blacklist is used by the first detection method to detect abnormal UA. The matching priority of each abnormal user agent in the blacklist is determined based on the matching frequency and / or the time when it was added to the blacklist.

10. The apparatus according to claim 9, wherein, The UA information detection module is used for: The target string is obtained by extracting the target quantity string from the UA information; The target string is matched against the blacklist. If the target string is matched in the blacklist, the UA information is determined to be an abnormal UA.

11. The apparatus according to claim 9, wherein, The update module is used for: Obtain traffic data containing the target string within the current statistical period to obtain the traffic dataset to be processed; Determine the proportion of abnormal traffic in the traffic dataset to be processed; If the proportion of abnormal traffic exceeds the proportion threshold and the target string is not in the blacklist, the target string will be updated to the blacklist required for the next statistical period.

12. The apparatus according to claim 9, wherein, The update module is used for: Retrieve User Agent (UA) information from the traffic data of pages accessed within the current statistical period; Determine the number of accessed pages whose User Agent (UA) information contains the target string; If the number of accessed pages containing the target string exceeds the page count threshold and the target string is not in the blacklist, the target string will be added to the blacklist required for the next statistical period.

13. The apparatus according to claim 9, wherein, The UA information detection module is used for: The UA information is compared with at least one syntactic feature in the abnormal UA feature set; If the UA information matches any syntactic feature, the UA information is determined to be an abnormal UA.

14. The apparatus according to claim 9, wherein the UA information detection module is used for: The target string is matched against the whitelist. If the target string matches the whitelist, the UA information is determined to be a normal UA.

15. The apparatus according to any one of claims 9-14, further comprising: The identifier acquisition module is used to acquire a unique identifier of the source of the traffic data; The first statistics module is used to calculate the abnormal UA carry-over rate corresponding to the unique identifier within a specified time period based on the unique identifier, as reference information for risk tracing.

16. The apparatus according to any one of claims 9-14, further comprising: The second statistics module is used to calculate the degree of missed detection by the anti-crawler strategy. An optimization module is used to determine the priority of the anti-crawler strategy that needs to be optimized based on the degree of missed detection.

17. An electronic device comprising: At least one processor; as well as A memory that is communicatively connected to at least one processor; wherein, The memory stores instructions that can be executed by at least one processor to enable the at least one processor to perform the method of any one of claims 1-8.

18. A non-transitory computer-readable storage medium storing computer instructions, wherein, Computer instructions are used to cause a computer to perform the method according to any one of claims 1-8.

19. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-8.

Images