An anti-crawler method, device, equipment and medium

Abstract

This application discloses an anti-crawler method, apparatus, device, and medium, relating to the field of network security technology. The method includes: real-time monitoring of the client to be monitored to obtain real-time user access data and real-time user behavior monitoring data; extracting user behavior feature data based on the real-time user access data and the real-time user behavior monitoring data; inputting the user behavior feature data into a preset target anti-crawler model; comparing the user behavior feature data according to preset judgment rules and the output result of the target anti-crawler model; determining a target response measure corresponding to the comparison result based on the obtained comparison result; executing the target response measure for anti-crawler processing; and saving the processing result to a local database. This enables intelligent identification of malicious crawlers and automated execution of corresponding anti-crawler measures.

Description

Technical Field

[0001] This invention relates to the field of network security technology, and in particular to an anti-crawler method, apparatus, device, and medium. Background Technology

[0002] With the rapid growth of online information, web crawler technology has been widely used in data collection. However, malicious crawlers may scrape large amounts of data for unfair competition, data resale, or other malicious purposes, seriously harming the interests of website owners and negatively impacting website performance, such as increased server load. Therefore, anti-crawler technology has emerged. Currently, anti-crawler technology is typically implemented through methods such as CAPTCHA verification and real-time traffic monitoring. However, these existing technologies have low accuracy in identifying malicious crawlers, resulting in high false positive and false negative rates. Furthermore, methods requiring user intervention, such as CAPTCHA verification, negatively affect the browsing experience of legitimate users, thus failing to effectively prevent web crawling.

[0003] In summary, how to intelligently identify malicious web crawlers and automatically execute corresponding anti-crawler measures is a pressing technical problem that needs to be solved. Summary of the Invention

[0004] In view of this, the purpose of this invention is to provide an anti-crawler method, apparatus, device, and medium capable of intelligently identifying malicious crawlers and automatically executing corresponding anti-crawler measures. The specific solution is as follows:

[0005] Firstly, this application provides an anti-crawler method applied to an anti-crawler system, comprising:

[0006] The client to be monitored is monitored in real time to obtain real-time user access data and real-time user behavior monitoring data, and user behavior feature data is extracted based on the real-time user access data and real-time user behavior monitoring data.

[0007] The user behavior feature data is input into a preset target anti-crawler model, and the user behavior feature data is compared according to the preset judgment rules and the output results of the target anti-crawler model. Then, the target response measures corresponding to the comparison results are determined based on the obtained comparison results.

[0008] The target response measures are executed to perform anti-crawler processing, and the processing results are saved to the local database.

[0009] Optionally, before performing real-time monitoring of the client to be monitored to obtain real-time user access data and real-time user behavior monitoring data, the method further includes:

[0010] User access data is collected from the client to be monitored, and user behavior is monitored to obtain user behavior monitoring data;

[0011] First behavioral feature data is extracted from the user access data, and second behavioral feature data is constructed using the user behavior monitoring data;

[0012] The first behavioral feature data and the second behavioral feature data are preprocessed to construct a target feature set based on the target feature data obtained after preprocessing.

[0013] The original model is trained based on the target feature set, and the resulting trained model is used as a preset target anti-crawler model.

[0014] Optionally, the preprocessing of the first behavioral feature data and the second behavioral feature data includes:

[0015] Duplicate data, missing data, and abnormal data are removed from the first behavioral feature data and the second behavioral feature data to obtain the feature data after removal.

[0016] The categorical features in the removed feature data are one-hot encoded, and the numerical features are normalized or standardized to obtain the target feature data.

[0017] Optionally, after executing the target response measures for anti-crawling processing and saving the processing results to the local database, the method further includes:

[0018] The target feature set is updated by adding new feature data based on a preset update frequency to obtain the updated feature set; wherein the added feature data is behavioral feature data obtained by extracting features from newly added user behavior monitoring data.

[0019] The updated feature set is used to evaluate the performance impact of the target anti-crawler model, so as to determine whether there is any feature data to be optimized in the updated feature set based on the evaluation results; the performance impact evaluation is an evaluation of the performance impact of the target anti-crawler model on the feature data in the target feature set.

[0020] If there are feature data to be optimized in the updated feature set, then the feature data to be optimized is removed from the updated feature set to obtain the target updated feature set;

[0021] The target anti-crawler model is updated and trained using the updated feature set of the target, so that the trained anti-crawler model is used as the current target anti-crawler model.

[0022] Optionally, the step of inputting the user behavior feature data into a preset target anti-crawler model, comparing the user behavior feature data according to preset judgment rules and the output of the target anti-crawler model, and then determining the target response measures corresponding to the comparison results based on the obtained comparison results includes:

[0023] The user behavior feature data is input into a preset target anti-crawler model to obtain the predicted anomaly probability distribution output by the target anti-crawler model corresponding to the user behavior feature data.

[0024] The predicted anomaly probability distribution is compared with a preset anomaly threshold to obtain the corresponding comparison results;

[0025] If the comparison result indicates that there is no target user behavior feature data higher than the preset abnormal threshold in the predicted abnormal probability distribution, then the user behavior feature data is determined to be normal user behavior feature data.

[0026] If the comparison result indicates that there are target user behavior feature data in the predicted abnormal probability distribution that are higher than the preset abnormal threshold, then the target user behavior feature data is regarded as suspicious user behavior feature data, and the target response measures corresponding to the suspicious user behavior feature data are determined in the local database.

[0027] Optionally, executing the target response measures for anti-crawling processing includes:

[0028] The target response measures are executed to ban or restrict access and limit the request rate of the suspicious users corresponding to the suspicious user behavior feature data;

[0029] Alternatively, an alert operation can be performed on the suspicious user, and after receiving the review result corresponding to the alert operation, the target anti-crawler model can be trained based on the review result.

[0030] Optionally, the anti-crawler method further includes:

[0031] Analyze the historical processing results stored in the local database to obtain the trend characteristics of malicious crawler behavior and determine whether there are any new malicious crawler attack patterns.

[0032] If the newly added malicious crawler attack mode exists, the preset judgment rules and the response measures currently stored in the local database will be updated based on the malicious behavior trend characteristics and the newly added malicious crawler attack mode.

[0033] If the newly added malicious crawler attack mode does not exist, the preset judgment rules and the response measures currently stored in the local database are updated based on the malicious behavior trend characteristics.

[0034] Secondly, this application provides an anti-crawler device, applied to an anti-crawler system, comprising:

[0035] The feature extraction module is used to perform real-time monitoring of the client to be monitored in order to obtain real-time user access data and real-time user behavior monitoring data, and to extract user behavior feature data based on the real-time user access data and the real-time user behavior monitoring data.

[0036] The measure determination module is used to input the user behavior feature data into a preset target anti-crawler model, compare the user behavior feature data according to preset judgment rules and the output results of the target anti-crawler model, and then determine the target response measures corresponding to the comparison results based on the obtained comparison results.

[0037] The result saving module is used to execute the target response measures for anti-crawler processing and save the processing results to the local database.

[0038] Thirdly, this application provides an electronic device, comprising:

[0039] Memory, used to store computer programs;

[0040] A processor is used to execute the computer program to implement the aforementioned anti-crawler method.

[0041] Fourthly, this application provides a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the aforementioned anti-crawler method.

[0042] In this application, the client to be monitored is monitored in real time to obtain real-time user access data and real-time user behavior monitoring data. User behavior feature data is extracted based on the real-time user access data and real-time user behavior monitoring data. The user behavior feature data is input into a preset target anti-crawler model, and the user behavior feature data is compared according to preset judgment rules and the output results of the target anti-crawler model. Then, based on the comparison results, a corresponding target response measure is determined. The target response measure is executed to perform anti-crawler processing, and the processing result is saved to a local database. As can be seen from the above, this application monitors the client to be monitored in real time to obtain relevant real-time user access data and real-time user behavior monitoring data. User behavior feature data is extracted based on the obtained real-time user access data and real-time user behavior monitoring data. The extracted user behavior feature data is input into a preset target anti-crawler model, and the user behavior feature data is compared according to preset judgment rules and the output results of the target anti-crawler model. Based on the comparison results, corresponding target response measures are executed to perform anti-crawler processing, and the processing result is saved to a local database. In this way, by using the results output by the preset target anti-crawler model and the preset judgment rules to compare user behavior feature data through the above process of this application, malicious crawlers can be intelligently identified, which can effectively reduce the false judgment rate, improve the security of the website, and thus improve the access experience of normal users. At the same time, based on the comparison results, corresponding target response measures are executed to carry out anti-crawler processing, thereby realizing intelligent identification of malicious crawlers and automatic execution of corresponding anti-crawler measures. Attached Figure Description

[0043] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0044] Figure 1 This is a flowchart of an anti-crawler method disclosed in this application;

[0045] Figure 2 This is a flowchart illustrating an anti-crawler method disclosed in this application;

[0046] Figure 3 This is a flowchart of a specific anti-scraping method disclosed in this application;

[0047] Figure 4 This is a schematic diagram of an anti-crawler device disclosed in this application;

[0048] Figure 5 This is a structural diagram of an electronic device disclosed in this application. Detailed Implementation

[0049] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0050] Currently, anti-crawler technology is usually implemented through methods such as CAPTCHA verification and real-time traffic monitoring. However, these existing technologies have a low accuracy rate in identifying malicious crawlers, resulting in a high rate of false positives and false negatives. In addition, methods such as CAPTCHA verification that require user intervention will affect the access experience of normal users and cannot effectively achieve anti-crawler measures.

[0051] To overcome the aforementioned technical problems, this application provides an anti-crawler method to intelligently identify malicious crawlers and automatically execute corresponding anti-crawler measures.

[0052] See Figure 1 As shown, this embodiment of the invention discloses an anti-crawler method, applied to an anti-crawler system, comprising:

[0053] Step S11: Perform real-time monitoring on the client to be monitored to obtain real-time user access data and real-time user behavior monitoring data, and extract user behavior feature data based on the real-time user access data and the real-time user behavior monitoring data.

[0054] In this embodiment, real-time monitoring of the client under monitoring is used to obtain real-time user access data and real-time user behavior monitoring data from the client. User behavior feature data is then extracted using the obtained real-time user access data and real-time user behavior monitoring data. Specifically, this embodiment can collect user access data and use front-end scripts to monitor user behavior, such as mouse movement, clicking, and scrolling, while simultaneously monitoring the client under monitoring, to obtain real-time user access data and real-time user behavior monitoring data. User behavior features are then extracted using the obtained data. The access data includes, but is not limited to, request frequency, request path, user agent, and access time. The client under monitoring is any client that requires real-time monitoring for crawler detection and anti-crawler processing based on the detection results, such as game application clients, browser clients with integrated game programs, and instant messaging clients.

[0055] It is understood that this embodiment can monitor not only the client but also the terminal and the server for anti-crawler processing. The terminal includes, but is not limited to, smartphones, tablets, laptops, touchscreens, game consoles, personal computers (PCs), and personal digital assistants (PDAs); the server can be a standalone physical server or a server cluster or distributed system composed of multiple physical servers.

[0056] To achieve the function of intelligently identifying malicious web crawlers, this embodiment needs to train a classification model for web crawler identification before real-time monitoring of the client to be monitored. The process is as follows: collect user access data from the client to be monitored and monitor user behavior to obtain user behavior monitoring data; extract first behavioral feature data from the user access data and construct second behavioral feature data using the user behavior monitoring data; preprocess the first behavioral feature data and the second behavioral feature data to construct a target feature set based on the target feature data obtained after preprocessing; train the original model based on the target feature set to use the trained model as a preset target anti-crawler model. Specifically, user access data is continuously collected from the monitored client, recording relevant information for each user request, such as network protocol (i.e., Internet Protocol, IP) address, request time, request path, request method, user agent, etc. Simultaneously, front-end scripts are integrated to monitor user behavior, such as mouse movement trajectory, page dwell time, scrolling behavior, etc., to obtain user behavior monitoring data. After ensuring the collected data format is consistent, the user access data and user behavior monitoring data are saved to a local database for subsequent analysis. Then, the first behavioral feature data is extracted from the user access data in the local database, and the second behavioral feature data is constructed using the user behavior monitoring data. The first and second behavioral feature data are preprocessed to eliminate dimensional differences, ensuring the effectiveness of model training, and a target feature set is constructed based on the preprocessed target feature data. The first behavioral feature data includes, but is not limited to, user access frequency, user request interval, and page access depth. Constructing the second behavioral feature data using the user behavior monitoring data can be achieved by analyzing the user's regular access paths to obtain access patterns, calculating the number of user interactions on the page to obtain interaction features, or identifying abnormal behavioral features based on abnormally high request frequencies or user behavior monitoring data that accesses a large number of pages in a short period. Finally, the original model is trained using the target feature set, and the resulting trained model is used as the preset target anti-crawler model. It is understood that this embodiment can periodically compare the actual crawler behavior features with the feature data in the target feature set to optimize the extraction strategy of the first behavioral feature data and the construction strategy of the second behavioral feature data based on the comparison results, thereby improving the model's adaptability.

[0057] It should be noted that, to avoid situations where the first and second behavioral feature data contain data that affects data quality, such as duplicate data, missing data, or abnormal data, leading to poor performance of the final trained anti-crawler model and its inability to effectively identify malicious crawlers, and to ensure the stability of model training, the first and second behavioral feature data need to be preprocessed. The processing flow is as follows: Duplicate, missing, and abnormal data are removed from the first and second behavioral feature data to obtain the removed feature data; the categorical features in the removed feature data are one-hot encoded, and the numerical features are normalized or standardized to obtain the target feature data. In other words, this embodiment removes duplicate, missing, and abnormal data from the first and second behavioral feature data to ensure data quality, while performing one-hot encoding on the categorical features in the removed feature data and normalizing or standardizing the numerical features in the removed feature data to ensure the stability of model training, improve the performance of the final trained anti-crawler model, and effectively identify malicious crawlers.

[0058] In this embodiment, the process of training the original model based on the target feature set to obtain the trained model as the preset target anti-crawler model is as follows: A suitable machine learning algorithm is selected based on data characteristics and business requirements, and the target feature set is divided to obtain a training dataset, a validation dataset, and a test dataset; the original model is trained based on the training dataset and preset hyperparameters to improve model performance, and the model's performance is monitored using the validation dataset to prevent overfitting, ultimately obtaining the trained model; the generalization ability and stability of the model are evaluated based on the test dataset, preset evaluation metrics, and K-fold cross-validation to ensure the reliability of the results; based on the obtained evaluation results, the preset hyperparameters are optimized using grid search or random search methods, and the importance of the target feature set is evaluated using a feature selection algorithm to determine the most effective target features for malicious crawler identification; the trained model is trained using the optimized preset hyperparameters and the target features to obtain the target anti-crawler model. The machine learning algorithms include, but are not limited to, support vector machines, random forests, gradient boosting trees, and deep learning. The training dataset, validation dataset, and test dataset can be divided into the target feature set according to a 14:3:3 ratio. The preset evaluation metrics include, but are not limited to, accuracy, precision, recall, and F1 score. The feature selection algorithms include, but are not limited to, chi-square test and random forest feature importance algorithm. In this embodiment, a classification model for crawler identification is first trained based on user access data and user behavior monitoring data collected from the client to be monitored, to achieve the function of intelligently identifying malicious crawlers. Then, the client to be monitored is monitored in real time, and user behavior feature data is extracted based on the real-time user access data and real-time user behavior monitoring data obtained through monitoring. This allows for a more comprehensive acquisition of user behavior feature data, facilitating the improvement of the accuracy of subsequent intelligent identification of malicious crawlers.

[0059] Step S12: Input the user behavior feature data into the preset target anti-crawler model, compare the user behavior feature data according to the preset judgment rules and the output results of the target anti-crawler model, and then determine the target response measures corresponding to the comparison results based on the obtained comparison results.

[0060] In this embodiment, the user behavior feature data is input into a preset target anti-crawler model to obtain the output result of the target anti-crawler model. The user behavior feature data is then compared according to preset judgment rules and the output result. Based on the comparison result, the type of the user behavior feature data is determined, and corresponding target response measures are determined according to the type. The target response measures include, but are not limited to, allowing users to continue accessing the site, triggering CAPTCHAs, reducing access frequency, limiting access time, and temporarily or permanently blocking IP addresses. The types of user behavior feature data include, but are not limited to, suspicious user behavior feature data, normal user behavior feature data, and malicious crawler behavior feature data.

[0061] It is understood that this embodiment can periodically analyze the historical processing results stored in the local database to obtain the malicious behavior trend characteristics of malicious crawlers and determine whether there are any new malicious crawler attack patterns. If the new malicious crawler attack pattern exists, the preset judgment rules and the response measures currently stored in the local database are updated based on the malicious behavior trend characteristics and the new malicious crawler attack pattern. If the new malicious crawler attack pattern does not exist, the preset judgment rules and the response measures currently stored in the local database are updated based on the malicious behavior trend characteristics. Here, the malicious behavior trend characteristics refer to the continuous and regular changes in the activity of malicious crawlers over a period of time. In other words, by analyzing historical processing results recorded in the local database at a preset frequency, malicious crawler behavior trend characteristics are obtained. Based on these malicious behavior trend characteristics, the preset judgment rules and the response measures currently stored in the local database are updated, enabling more effective adjustment of anti-crawler strategies. Simultaneously, it determines whether any new malicious crawler attack patterns exist in the historical processing results. If such new malicious crawler attack patterns exist, it means that if a malicious crawler corresponding to the new malicious crawler attack pattern reappears, the current preset judgment rules and the response measures currently stored in the local database may not be able to effectively identify and process the malicious crawler. Therefore, the preset judgment rules and the response measures currently stored in the local database can be updated based on the malicious behavior trend characteristics and the new malicious crawler attack patterns. By determining the malicious behavior trend characteristics and whether any new malicious crawler attack patterns exist, anti-crawler strategies can be adjusted more effectively. Furthermore, by continuously updating the preset judgment rules and the response measures, the adaptability and protection capabilities of the anti-crawler system are improved, and the false negative rate of the anti-crawler system is reduced. In this way, this embodiment inputs the user behavior feature data into a preset target anti-crawler model to intelligently identify the user behavior feature data, and then determines the corresponding target response measures based on the output of the target anti-crawler model in order to carry out anti-crawler processing. At the same time, by analyzing the recorded historical processing results, the preset judgment rules and the saved response measures are continuously updated, which improves the adaptability of the anti-crawler system, enables more flexible identification of user behavior feature data, and reduces the possibility of misjudgment.

[0062] Step S13: Execute the target response measures to perform anti-crawler processing and save the processing results to the local database.

[0063] In this embodiment, anti-crawler processing is performed by executing the determined target response measures, and the processing results are saved to a local database. Specifically, the processing flow for executing the target response measures to perform anti-crawler processing is as follows: The target response measures are executed to ban or restrict access and limit the request rate of suspicious users corresponding to suspicious user behavior feature data; or, an alarm operation is performed on the suspicious users, and after receiving the review result corresponding to the alarm operation, the target anti-crawler model is trained based on the review result. That is, if the comparison result indicates that the user behavior feature data is suspicious user behavior feature data, then the suspicious users corresponding to the suspicious user behavior feature data are banned or restricted in access and limited in request rate; or, an alarm message is generated for the suspicious users, and the alarm message is sent to the terminal where the relevant system administrators are located, so that the relevant system administrators can review the suspicious users and label the suspicious user behavior feature data, and feed back the review result and the labeled data to the anti-crawler system. Finally, the target anti-crawler model is updated and trained based on the received review result and the labeled data, forming a closed-loop feedback mechanism. In addition, the suspicious user can be required to undergo a fully Automated Public Turing Test to Tell Humans Apart (CAPTCHA) to verify their identity. It should be noted that the aforementioned alerting operation for the suspicious user is performed only after determining that the suspiciousness level of the user exceeds a preset threshold. Specifically, if the suspicious user is highly suspicious, an alert is issued, and the received review results are used to determine whether the suspicious user is a malicious web crawler, and the target anti-crawler model is updated and trained. The criteria for determining whether a user is highly suspicious can be that the difference between the output of the target anti-crawler model and the preset anomaly threshold exceeds a preset quantity threshold, or that the target anti-crawler model cannot identify the suspicious user's behavioral characteristic data.

[0064] It should be noted that, in order to improve the adaptability and reliability of the anti-crawler method, this embodiment can continuously optimize and update the anti-crawler method after saving the processing results to the local database. The processing flow is as follows: The target feature set is updated by adding new feature data based on a preset update frequency to obtain an updated feature set; wherein the added feature data is behavioral feature data obtained by feature extraction from newly added user behavior monitoring data; the performance impact of the updated feature set is evaluated on the target anti-crawler model to determine whether there is any feature data to be optimized in the updated feature set; the performance impact evaluation is an evaluation of the performance impact of the feature data in the target feature set on the target anti-crawler model; if there is any feature data to be optimized in the updated feature set, the feature data to be optimized is removed from the updated feature set to obtain the target updated feature set; the target anti-crawler model is updated and trained using the target updated feature set to use the trained anti-crawler model as the current target anti-crawler model. In other words, this embodiment periodically updates the target feature set using newly collected feature data to train and update the target anti-crawler model, adjusting its decision boundary to ensure the model maintains high accuracy and improves its recognition ability. Simultaneously, it periodically evaluates the impact of features on model performance to identify features in the target feature set that require optimization—features that are ineffective in new environments or whose importance for crawler identification has decreased. These unoptimized features are removed from the target feature set to enhance the model's adaptability. Figure 2 The diagram shown is a schematic of an overall anti-scraping method.

[0065] Furthermore, to ensure that the target anti-crawler model can quickly adapt to new crawler behaviors without consuming excessive computing resources, this embodiment can use an incremental learning algorithm to update the existing target anti-crawler model when new feature data arrives, avoiding training from scratch and consuming unnecessary computing resources. In this way, this embodiment performs anti-crawler processing based on target response measures determined by comparison results and saves the processing results to a local database. This allows for continuous monitoring of malicious crawler behavior trends based on the saved processing results, enabling dynamic adjustments to the anti-crawler strategy according to changes in crawler technology and improving the target anti-crawler model's sensitivity to new malicious crawler attack patterns.

[0066] As can be seen from the above, the embodiments of this application perform real-time monitoring of the client to be monitored to obtain relevant real-time user access data and real-time user behavior monitoring data, and extract user behavior feature data based on the obtained real-time user access data and real-time user behavior monitoring data. The extracted user behavior feature data is input into a preset target anti-crawler model, and the user behavior feature data is compared based on preset judgment rules and the output results of the target anti-crawler model. Based on the comparison results, corresponding target response measures are executed to perform anti-crawler processing, and the processing results are saved to a local database. In this way, through the above process of the embodiments of this application, on the one hand, the client to be monitored is monitored in real time, and user behavior feature data is extracted based on the real-time user access data and real-time user behavior monitoring data obtained through monitoring. This enables a more comprehensive acquisition of user behavior feature data, which facilitates the improvement of the accuracy of subsequent intelligent identification of malicious crawlers. On the other hand, the results output by the preset target anti-crawler model and the preset judgment rules are compared with the user behavior feature data to achieve intelligent identification of malicious crawlers, which can effectively reduce the false judgment rate, improve the security of the website, and thus improve the access experience of normal users. Furthermore, based on the comparison results, corresponding target response measures are executed to perform anti-crawler processing, and the processing results are saved to the local database so that the behavior trend characteristics of malicious crawlers can be understood in a timely manner based on the saved processing results. This allows for dynamic adjustment of anti-crawler strategies according to changes in crawler technology, improving the sensitivity of the target anti-crawler model to new malicious crawler attack modes, thereby achieving intelligent identification of malicious crawlers and automatically executing corresponding anti-crawler measures.

[0067] As can be seen from the foregoing embodiments, the method of this application, after inputting user behavior feature data into a preset target anti-crawler model, can compare the user behavior feature data according to the output of the target anti-crawler model and preset judgment rules, so as to determine the corresponding target response measures based on the comparison results. Therefore, this embodiment provides a detailed explanation of how to compare user behavior feature data and determine the corresponding target response measures. See [link to documentation]. Figure 3 As shown, this embodiment of the invention discloses an anti-crawler method, applied to an anti-crawler system, comprising:

[0068] Step S21: Perform real-time monitoring on the client to be monitored to obtain real-time user access data and real-time user behavior monitoring data, and extract user behavior feature data based on the real-time user access data and the real-time user behavior monitoring data.

[0069] Step S22: Input the user behavior feature data into a preset target anti-crawler model to obtain the predicted anomaly probability distribution output by the target anti-crawler model corresponding to the user behavior feature data.

[0070] In this embodiment, the user behavior feature data is input into a preset target anti-crawler model to obtain the output of the target anti-crawler model, namely, the predicted anomaly probability distribution corresponding to the user behavior feature data. It is understood that the predicted anomaly probability distribution represents the likelihood that the user behavior feature data is identified as abnormal behavior feature data. Using the predicted anomaly probability distribution in conjunction with preset judgment rules helps determine whether the user behavior feature data is malicious crawler data, thereby enabling anti-crawler processing and improving network security.

[0071] Step S23: Compare the predicted anomaly probability distribution with the preset anomaly threshold to obtain the corresponding comparison results.

[0072] In this embodiment, the predicted anomaly probability distribution corresponding to the user behavior feature data output by the target anti-crawler model is compared with a preset anomaly threshold to obtain a comparison result of whether there is target user behavior feature data in the predicted anomaly probability distribution that is higher than the preset anomaly threshold. The preset anomaly threshold can be pre-set by the system administrator based on factors such as business needs, security policies, and model performance, and is usually set to one or more specific values ​​to distinguish between normal user behavior feature data and suspicious user behavior feature data. For example, if the request frequency in the user behavior feature data exceeds the preset anomaly threshold, or the number of user page visits exceeds the preset anomaly threshold in a short period of time, the user behavior feature data can be determined as suspicious user behavior feature data. Specifically, when comparing the predicted anomaly probability distribution with the preset anomaly threshold, this embodiment can iterate through each probability value in the predicted anomaly probability distribution and compare each probability value with the preset anomaly threshold. If there is a target probability value higher than the preset anomaly threshold, the user behavior feature data corresponding to the target probability value can be determined as suspicious user behavior feature data.

[0073] It is understandable that if the preset anomaly threshold is set too high, some genuinely suspicious user behavior data may be missed, making it difficult to guarantee website security. Conversely, if the preset anomaly threshold is set too low, the false positive rate may be high, misclassifying normal user behavior data as suspicious. Therefore, setting the preset anomaly threshold is crucial. Furthermore, as web crawler technology continues to improve and new types of crawlers are constantly emerging, this embodiment can periodically adjust and optimize the preset anomaly threshold based on user feedback, business needs, and other factors to ensure the accuracy of the anti-crawler system. In this way, this embodiment compares the predicted anomaly probability distribution with the preset anomaly threshold to obtain corresponding comparison results. Based on these results, the user behavior data can be judged, and corresponding target response measures can be executed. Simultaneously, setting the preset anomaly threshold for comparison with the predicted anomaly probability distribution output by the target anti-crawler model enables intelligent identification of malicious crawlers, ensuring website security.

[0074] Step S24: If the comparison result indicates that there is no target user behavior feature data higher than the preset abnormal threshold in the predicted abnormal probability distribution, then the user behavior feature data is determined to be normal user behavior feature data.

[0075] In this embodiment, if the comparison result indicates that there is no target user behavior feature data higher than the preset abnormal threshold in the predicted abnormal probability distribution, it means that the user behavior feature data does not have abnormal behavior. The user behavior feature data can be judged as normal user behavior feature data, and the normal user corresponding to the normal user behavior feature data is allowed to continue to access the site. By timely judging the user behavior feature data and executing corresponding response measures, the user's access experience can be improved.

[0076] Step S25: If the comparison result indicates that there are target user behavior feature data in the predicted abnormal probability distribution that are higher than the preset abnormal threshold, then the target user behavior feature data is regarded as suspicious user behavior feature data, and the target response measures corresponding to the suspicious user behavior feature data are determined in the local database.

[0077] In this embodiment, if the comparison result indicates that there is target user behavior feature data in the predicted anomaly probability distribution that is higher than the preset anomaly threshold, it indicates that the target user behavior feature data has abnormal behavior. The target user behavior feature data can then be designated as suspicious user behavior feature data, and corresponding target response measures can be determined in the local database. For example, the suspicious user corresponding to the suspicious user behavior feature data can be required to undergo CAPTCHA verification to confirm their identity; for IPs corresponding to frequently accessed suspicious user behavior feature data, temporary access restrictions can be implemented to reduce their request rate; for suspicious user behavior feature data confirmed as malicious crawlers, the IPs of the suspicious user behavior feature data can be added to a blacklist to prevent their subsequent access behavior. In this way, after designating the target user behavior feature data as suspicious user behavior feature data, this embodiment can determine corresponding target response measures from the local database based on the analysis of the suspicious user behavior feature data, thereby improving the effectiveness of anti-crawler processing.

[0078] Step S26: Execute the target response measures to perform anti-crawler processing, and save the processing results to the local database.

[0079] The specific implementation process of steps S21 and S26 can be referred to the content of the foregoing embodiments, and will not be repeated here.

[0080] As can be seen from the above, the embodiments of this application perform real-time monitoring of the client to be monitored to obtain relevant real-time user access data and real-time user behavior monitoring data. Based on the obtained real-time user access data and real-time user behavior monitoring data, user behavior feature data is extracted. The extracted user behavior feature data is input into a preset target anti-crawler model to obtain the predicted anomaly probability distribution corresponding to the user behavior feature data output by the target anti-crawler model. The predicted anomaly probability distribution is compared with a preset anomaly threshold. Based on the comparison result, the user behavior feature data is judged, and corresponding target response measures are executed for anti-crawler processing. The processing result is saved to a local database. In this way, the process described in this application embodiment, on the one hand, by inputting user behavior feature data into a preset target anti-crawler model to obtain the output predicted anomaly probability distribution, helps to determine whether the user behavior feature data is malicious crawler, and then performs anti-crawler processing to improve network security; on the other hand, by comparing the predicted anomaly probability distribution with a preset anomaly threshold, it is possible to intelligently identify malicious crawlers and ensure website security; and on the other hand, after judging the user behavior feature data based on the comparison results, it is possible to determine the corresponding target response measures in a targeted manner, which can improve the effectiveness of anti-crawler processing and the user's access experience, thereby realizing intelligent identification of malicious crawlers and automatic execution of corresponding anti-crawler measures.

[0081] Accordingly, see Figure 4 As shown in the illustration, this application also provides an anti-crawler device, applied to an anti-crawler system, comprising:

[0082] The feature extraction module 11 is used to perform real-time monitoring of the client to be monitored in order to obtain real-time user access data and real-time user behavior monitoring data, and extract user behavior feature data based on the real-time user access data and the real-time user behavior monitoring data.

[0083] The measure determination module 12 is used to input the user behavior feature data into a preset target anti-crawler model, compare the user behavior feature data according to the preset judgment rules and the output results of the target anti-crawler model, and then determine the target response measures corresponding to the comparison results based on the obtained comparison results.

[0084] The result saving module 13 is used to execute the target response measures for anti-crawler processing and save the processing results to the local database.

[0085] As can be seen from the above, this application embodiment performs real-time monitoring of the client to be monitored to obtain relevant real-time user access data and real-time user behavior monitoring data. Based on the obtained real-time user access data and real-time user behavior monitoring data, user behavior feature data is extracted. The extracted user behavior feature data is input into a preset target anti-crawler model. The user behavior feature data is compared based on preset judgment rules and the output results of the target anti-crawler model. Based on the comparison results, corresponding target response measures are executed for anti-crawler processing, and the processing results are saved to a local database. In this way, by using the above process of this application embodiment to compare user behavior feature data with the output results of the preset target anti-crawler model and preset judgment rules, intelligent identification of malicious crawlers can be achieved, effectively reducing the false positive rate, improving website security, and thus improving the access experience of normal users. Simultaneously, by executing corresponding target response measures based on the comparison results for anti-crawler processing, intelligent identification of malicious crawlers and automated execution of corresponding anti-crawler measures are achieved.

[0086] In some specific embodiments, the feature extraction module 11 may further include:

[0087] The behavior monitoring submodule is used to collect user access data from the client to be monitored and to monitor user behavior to obtain user behavior monitoring data.

[0088] The data construction submodule is used to extract first behavioral feature data from the user access data and construct second behavioral feature data using the user behavior monitoring data.

[0089] The data preprocessing submodule is used to preprocess the first behavioral feature data and the second behavioral feature data to construct a target feature set based on the target feature data obtained after preprocessing.

[0090] The model training submodule is used to train the original model based on the target feature set so that the trained model can be used as a preset target anti-crawler model.

[0091] In some specific embodiments, the data preprocessing submodule may specifically include:

[0092] The data removal unit is used to remove duplicate data, missing data and abnormal data from the first behavioral feature data and the second behavioral feature data to obtain the feature data after removal.

[0093] The data processing unit is used to perform one-hot encoding on the categorical features in the removed feature data and to normalize or standardize the numerical features to obtain the target feature data.

[0094] In some specific embodiments, the anti-crawler device may further include:

[0095] The feature set update unit is used to update the target feature set with newly added feature data based on a preset update frequency to obtain the updated feature set; wherein the newly added feature data is behavioral feature data obtained by extracting features from newly added user behavior monitoring data;

[0096] The model evaluation unit is used to evaluate the performance impact of the target anti-crawler model using the updated feature set, so as to determine whether there is feature data to be optimized in the updated feature set based on the evaluation results; the performance impact evaluation is an evaluation of the performance impact of the target anti-crawler model on the feature data in the target feature set.

[0097] The data removal unit is used to remove the feature data to be optimized from the updated feature set if there is feature data to be optimized in the updated feature set, so as to obtain the target updated feature set.

[0098] The model update unit is used to update and train the target anti-crawler model using the updated feature set of the target, so as to use the trained anti-crawler model as the current target anti-crawler model.

[0099] In some specific embodiments, the measure determination module 12 may specifically include:

[0100] The data input unit is used to input the user behavior feature data into a preset target anti-crawler model to obtain the predicted anomaly probability distribution output by the target anti-crawler model corresponding to the user behavior feature data.

[0101] An anomaly comparison unit is used to compare the predicted anomaly probability distribution with a preset anomaly threshold to obtain the corresponding comparison result.

[0102] The condition determination unit is used to determine that the user behavior feature data is normal user behavior feature data if the comparison result indicates that there is no target user behavior feature data higher than the preset abnormal threshold in the predicted abnormal probability distribution.

[0103] The measure determination unit is used to determine the target user behavior feature data as suspicious user behavior feature data if the comparison result indicates that there is target user behavior feature data in the predicted abnormal probability distribution that is higher than the preset abnormal threshold, and to determine the target response measure corresponding to the suspicious user behavior feature data in the local database.

[0104] In some specific embodiments, the result storage module 13 may specifically include:

[0105] The measure execution unit is used to execute the target response measures to ban or restrict access and limit the request rate of the suspicious users corresponding to the suspicious user behavior feature data;

[0106] Alternatively, a model training unit is used to perform alert operations on the suspicious user, and after receiving the review result corresponding to the alert operation, to train the target anti-crawler model based on the review result.

[0107] In some specific embodiments, the anti-crawler device may further include:

[0108] The results analysis unit is used to analyze the historical processing results stored in the local database to obtain the malicious behavior trend characteristics of malicious crawlers and determine whether there are any new malicious crawler attack patterns.

[0109] The first measure update unit is used to update the preset judgment rules and the response measures currently stored in the local database based on the malicious behavior trend characteristics and the new malicious crawler attack mode if the new malicious crawler attack mode exists.

[0110] The second measure update unit is used to update the preset judgment rules and the response measures currently stored in the local database based on the malicious behavior trend characteristics if the newly added malicious crawler attack mode does not exist.

[0111] Furthermore, embodiments of this application also disclose an electronic device, Figure 5 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content of the diagram should not be construed as limiting the scope of this application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the anti-crawler method disclosed in any of the foregoing embodiments. Furthermore, the electronic device 20 in this embodiment may specifically be a computer.

[0112] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0113] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk, or optical disk, etc. The resources stored thereon can include an operating system 221, computer programs 222, etc., and the storage method can be temporary storage or permanent storage.

[0114] The operating system 221 is used to manage and control the various hardware devices on the electronic device 20 and the computer program 222, which may be Windows Server, Netware, Unix, Linux, etc. In addition to including computer programs capable of performing the anti-crawler methods executed by the electronic device 20 as disclosed in any of the foregoing embodiments, the computer program 222 may further include computer programs capable of performing other specific tasks.

[0115] Furthermore, this application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the aforementioned anti-crawler method. Specific steps of this method can be found in the corresponding content disclosed in the foregoing embodiments, and will not be repeated here.

[0116] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0117] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0118] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0119] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0120] The technical solutions provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. An anti-crawler method, characterized in that, The application is applied to an anti-crawler system, comprising: real-time monitoring of a to-be-monitored client to obtain real-time user access data and real-time user behavior monitoring data, and extracting user behavior feature data based on the real-time user access data and the real-time user behavior monitoring data; inputting the user behavior feature data into a preset target anti-crawler model, comparing the user behavior feature data according to a preset judgment rule and an output result of the target anti-crawler model, and then determining a target response measure corresponding to the comparison result based on the comparison result obtained; performing the target response measure for anti-crawler processing, and saving a processing result to a local database; wherein, before the real-time monitoring of the to-be-monitored client to obtain the real-time user access data and the real-time user behavior monitoring data, the method further comprises: collecting user access data from the to-be-monitored client, and monitoring user behavior to obtain user behavior monitoring data; extracting first behavior feature data from the user access data, and constructing second behavior feature data by using the user behavior monitoring data; preprocessing the first behavior feature data and the second behavior feature data to construct a target feature set based on target feature data obtained after preprocessing; training an original model based on the target feature set to take a trained model obtained as a preset target anti-crawler model; wherein, the preprocessing of the first behavior feature data and the second behavior feature data comprises: eliminating repeated data, missing data and abnormal data in the first behavior feature data and the second behavior feature data to obtain eliminated feature data; one-hot encoding classification features in the eliminated feature data, and normalizing or standardizing numerical value features to obtain target feature data; wherein, after the performing of the target response measure for anti-crawler processing and the saving of the processing result to the local database, the method further comprises: updating the target feature set by adding new feature data based on a preset update frequency to obtain an updated feature set; wherein the new feature data is behavior feature data obtained by feature extraction on new user behavior monitoring data; performing performance impact evaluation on the target anti-crawler model through the updated feature set to determine whether there is to-be-optimized feature data in the updated feature set according to the evaluation result; the performance impact evaluation is an evaluation of performance impact test on the target anti-crawler model by using feature data in the target feature set; if there is to-be-optimized feature data in the updated feature set, removing the to-be-optimized feature data from the updated feature set to obtain a target updated feature set; updating and training the target anti-crawler model through the target updated feature set to take a trained anti-crawler model obtained as a current target anti-crawler model; wherein, the anti-crawler method further comprises: analyzing historical processing results saved in the local database to obtain malicious behavior trend features of malicious crawlers, and determining whether there is a new malicious crawler attack mode. If the new malicious crawler attack mode exists, the preset judgment rule and the currently saved response measure in the local database are updated based on the malicious behavior trend feature and the new malicious crawler attack mode. If the new malicious crawler attack mode does not exist, the preset judgment rule and the currently saved response measure in the local database are updated based on the malicious behavior trend feature.

2. The anti-crawler method of claim 1, wherein, The user behavior feature data is input into a preset target anti-crawler model, and the user behavior feature data is compared according to a preset judgment rule and an output result of the target anti-crawler model, and then a target response measure corresponding to the comparison result is determined based on the comparison result obtained. The user behavior feature data is input into a preset target anti-crawler model to obtain a predicted abnormal probability distribution corresponding to the user behavior feature data output by the target anti-crawler model. The predicted abnormal probability distribution is compared with a preset abnormal threshold to obtain a corresponding comparison result. If the comparison result indicates that there is no target user behavior feature data higher than the preset abnormal threshold in the predicted abnormal probability distribution, the user behavior feature data is determined to be normal user behavior feature data. If the comparison result indicates that there is target user behavior feature data higher than the preset abnormal threshold in the predicted abnormal probability distribution, the target user behavior feature data is regarded as suspicious user behavior feature data, and a target response measure corresponding to the suspicious user behavior feature data is determined in the local database.

3. The anti-crawler method of claim 2, wherein, The target response measure is executed for anti-crawler processing, including: The target response measure is executed to perform user ban or access restriction and request rate limitation on a suspicious user corresponding to the suspicious user behavior feature data. Or, an alarm operation is performed on the suspicious user, and after receiving an audit result corresponding to the alarm operation, the target anti-crawler model is trained based on the audit result.

4. A reverse crawler device, characterized by, Applied to an anti-crawler system, including: A feature extraction module is configured to monitor a to-be-monitored client in real time to obtain real-time user access data and real-time user behavior monitoring data, and extract user behavior feature data based on the real-time user access data and the real-time user behavior monitoring data. A measure determination module is configured to input the user behavior feature data into a preset target anti-crawler model, compare the user behavior feature data according to a preset judgment rule and an output result of the target anti-crawler model, and then determine a target response measure corresponding to the comparison result based on the comparison result obtained. A result saving module is configured to execute the target response measure for anti-crawler processing, and save a processing result to a local database. The feature extraction module includes: A behavior monitoring submodule is configured to collect user access data from the to-be-monitored client, and monitor user behavior to obtain user behavior monitoring data. The data construction submodule is configured to extract first behavior feature data from the user access data and construct second behavior feature data by using the user behavior monitoring data; The data preprocessing submodule is configured to preprocess the first behavior feature data and the second behavior feature data, and construct a target feature set based on target feature data obtained after preprocessing; The model training submodule is configured to train an original model based on the target feature set to obtain a trained model as a preset target anti-crawler model; The data preprocessing submodule includes: The data elimination unit is configured to eliminate duplicate data, missing data, and abnormal data in the first behavior feature data and the second behavior feature data to obtain eliminated feature data; The data processing unit is configured to perform one-hot encoding on classification features in the eliminated feature data and perform normalization or standardization processing on numerical features to obtain target feature data; The anti-crawler device further includes: The feature set updating unit is configured to update the target feature set by adding new feature data based on a preset update frequency to obtain an updated feature set; the new feature data is behavior feature data obtained by extracting features from new user behavior monitoring data; The model evaluation unit is configured to perform performance impact evaluation on the target anti-crawler model by using the updated feature set to determine whether there is to-be-optimized feature data in the updated feature set according to the evaluation result; the performance impact evaluation is an evaluation of performance impact testing on the target anti-crawler model by using feature data in the target feature set; The data removal unit is configured to remove the to-be-optimized feature data from the updated feature set to obtain a target updated feature set if there is to-be-optimized feature data in the updated feature set; The model updating unit is configured to update train the target anti-crawler model by using the target updated feature set to obtain a trained anti-crawler model as a current target anti-crawler model; The anti-crawler device further includes: The result analysis unit is configured to analyze historical processing results saved in the local database to obtain malicious behavior trend features of malicious crawlers and determine whether there is a new malicious crawler attack mode; The first measure updating unit is configured to update the preset determination rule and the currently saved response measure in the local database based on the malicious behavior trend features and the new malicious crawler attack mode if the new malicious crawler attack mode exists; The second measure updating unit is configured to update the preset determination rule and the currently saved response measure in the local database based on the malicious behavior trend features if the new malicious crawler attack mode does not exist.

5. An electronic device, comprising: The memory is configured to save a computer program; The processor is configured to execute the computer program to implement the anti-crawler method of any one of claims 1 to 3. ​ 6. A computer-readable storage medium, characterized in that, A computer program product for saving a computer program; wherein the computer program, when executed by a processor, implements the anti-crawler method according to any one of claims 1 to 3.

Images