Anti-crawler method and device, equipment and medium

Abstract

The invention discloses an anti-crawler method and device, equipment and a medium, and relates to the technical field of network security, and the method comprises the steps: carrying out the real-time monitoring of a to-be-monitored client, so as to obtain real-time user access data and real-time user behavior monitoring data, extracting user behavior characteristic data based on the real-time user access data and the real-time user behavior monitoring data; inputting the user behavior characteristic data into a preset target anti-crawler model, comparing the user behavior characteristic data according to a preset judgment rule and an output result of the target anti-crawler model, and then determining a target response measure corresponding to an obtained comparison result based on the obtained comparison result; and executing the target response measure to perform anti-crawler processing, and storing a processing result to a local database. Therefore, malicious crawlers can be intelligently identified, and corresponding anti-crawler measures can be automatically executed.

Description

Technical Field

[0001] The present invention relates to the field of network security technology, and in particular to an anti-crawler method, device, equipment and medium. Background Art

[0002] With the rapid growth of network information, web crawler technology has been widely used in the field of data collection. However, malicious crawlers may grab a large amount of data for unfair competition, data reselling or other malicious purposes, seriously damaging the interests of website owners, and causing negative impacts such as increased server load on the website. Therefore, anti-crawler technology has emerged. At present, anti-crawler technology is usually implemented through methods such as verification code verification and real-time traffic monitoring. However, these existing technologies have low recognition accuracy for malicious crawlers, resulting in high misjudgment rate and missed judgment rate of malicious crawlers. At the same time, methods such as verification code verification that require user intervention will affect the access experience of normal users and cannot effectively implement anti-crawler.

[0003] From the above, it can be seen that how to intelligently identify malicious crawlers and automatically execute corresponding anti-crawler measures is a technical problem that needs to be solved urgently. Summary of the invention

[0004] In view of this, the purpose of the present invention is to provide an anti-crawler method, device, equipment and medium, which can realize intelligent identification of malicious crawlers and automatically execute corresponding anti-crawler measures. The specific scheme is as follows:

[0005] In a first aspect, the present application provides an anti-crawler method, which is applied to an anti-crawler system, comprising:

[0006] Performing real-time monitoring on the monitored client to obtain real-time user access data and real-time user behavior monitoring data, and extracting user behavior feature data based on the real-time user access data and the real-time user behavior monitoring data;

[0007] Inputting the user behavior feature data into a preset target anti-crawler model, and comparing the user behavior feature data according to a preset determination rule and an output result of the target anti-crawler model, and then determining a target response measure corresponding to the comparison result based on the obtained comparison result;

[0008] The target response measures are executed to perform anti-crawler processing, and the processing results are saved to a local database.

[0009] Optionally, before performing real-time monitoring on the client to be monitored to obtain real-time user access data and real-time user behavior monitoring data, the method further includes:

[0010] Collecting user access data from the client to be monitored, and monitoring user behavior to obtain user behavior monitoring data;

[0011] Extracting first behavior characteristic data from the user access data, and constructing second behavior characteristic data using the user behavior monitoring data;

[0012] Preprocessing the first behavior feature data and the second behavior feature data to construct a target feature set based on the target feature data obtained after the preprocessing;

[0013] The original model is trained based on the target feature set to use the obtained trained model as a preset target anti-crawler model.

[0014] Optionally, the preprocessing the first behavior characteristic data and the second behavior characteristic data includes:

[0015] Eliminate duplicate data, missing data, and abnormal data in the first behavior characteristic data and the second behavior characteristic data to obtain eliminated characteristic data;

[0016] The categorical features in the removed feature data are one-hot encoded, and the numerical features are normalized or standardized to obtain target feature data.

[0017] Optionally, after executing the target response measure to perform anti-crawler processing and saving the processing result to a local database, the method further includes:

[0018] Based on a preset update frequency, the target feature set is updated by adding feature data to obtain an updated feature set; wherein the newly added feature data is behavior feature data obtained by extracting features from newly added user behavior monitoring data;

[0019] A performance impact assessment is performed on the target anti-crawler model through the updated feature set, so as to determine whether there is feature data to be optimized in the updated feature set according to the assessment result; the performance impact assessment is an assessment of a performance impact test performed on the target anti-crawler model through the feature data in the target feature set;

[0020] If there is feature data to be optimized in the updated feature set, the feature data to be optimized is removed from the updated feature set to obtain a target updated feature set;

[0021] The target anti-crawler model is updated and trained using the target updated feature set, so that the obtained trained anti-crawler model is used as the current target anti-crawler model.

[0022] Optionally, the inputting the user behavior feature data into a preset target anti-crawler model, comparing the user behavior feature data according to a preset determination rule and an output result of the target anti-crawler model, and then determining a target response measure corresponding to the comparison result based on the obtained comparison result, includes:

[0023] Inputting the user behavior feature data into a preset target anti-crawler model to obtain a predicted abnormal probability distribution corresponding to the user behavior feature data output by the target anti-crawler model;

[0024] Comparing the predicted abnormal probability distribution with a preset abnormal threshold to obtain a corresponding comparison result;

[0025] If the comparison result indicates that there is no target user behavior feature data higher than the preset abnormality threshold in the predicted abnormal probability distribution, then the user behavior feature data is determined to be normal user behavior feature data;

[0026] If the comparison result indicates that there is target user behavior feature data higher than the preset abnormality threshold in the predicted abnormal probability distribution, the target user behavior feature data is used as suspicious user behavior feature data, and a target response measure corresponding to the suspicious user behavior feature data is determined in the local database.

[0027] Optionally, executing the target response measure to perform anti-crawler processing includes:

[0028] Executing the target response measures to ban or restrict access and limit request rate for the suspicious user corresponding to the suspicious user behavior feature data;

[0029] Or, perform an alarm operation on the suspicious user, and after receiving the audit result corresponding to the alarm operation, train the target anti-crawler model based on the audit result.

[0030] Optionally, the anti-crawler method further includes:

[0031] Analyze the historical processing results saved in the local database to obtain the malicious behavior trend characteristics of malicious crawlers and determine whether there are new malicious crawler attack patterns;

[0032] If the newly added malicious crawler attack pattern exists, the preset judgment rule and the response measures currently stored in the local database are updated based on the malicious behavior trend characteristics and the newly added malicious crawler attack pattern;

[0033] If the newly added malicious crawler attack pattern does not exist, the preset judgment rules and the response measures currently stored in the local database are updated based on the malicious behavior trend characteristics.

[0034] In a second aspect, the present application provides an anti-crawler device, which is applied to an anti-crawler system, including:

[0035] A feature extraction module is used to perform real-time monitoring on the monitored client to obtain real-time user access data and real-time user behavior monitoring data, and extract user behavior feature data based on the real-time user access data and the real-time user behavior monitoring data;

[0036] A measure determination module, used for inputting the user behavior feature data into a preset target anti-crawler model, and comparing the user behavior feature data according to a preset determination rule and an output result of the target anti-crawler model, and then determining a target response measure corresponding to the comparison result based on the comparison result;

[0037] The result saving module is used to execute the target response measures to perform anti-crawler processing and save the processing results to a local database.

[0038] In a third aspect, the present application provides an electronic device, including:

[0039] Memory, used to store computer programs;

[0040] A processor is used to execute the computer program to implement the aforementioned anti-crawler method.

[0041] In a fourth aspect, the present application provides a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, the aforementioned anti-crawler method is implemented.

[0042] In the present application, the monitored client is monitored in real time to obtain real-time user access data and real-time user behavior monitoring data, and user behavior feature data is extracted based on the real-time user access data and the real-time user behavior monitoring data; the user behavior feature data is input into a preset target anti-crawler model, and the user behavior feature data is compared according to the preset judgment rule and the output result of the target anti-crawler model, and then the target response measures corresponding to the comparison result are determined based on the obtained comparison result; the target response measures are executed to perform anti-crawler processing, and the processing results are saved to a local database. As can be seen from the above, the present application is to monitor the monitored client in real time to obtain relevant real-time user access data and real-time user behavior monitoring data, and user behavior feature data is extracted based on the acquired real-time user access data and the real-time user behavior monitoring data, and the extracted user behavior feature data is input into a preset target anti-crawler model, and the user behavior feature data is compared based on the preset judgment rule and the output result of the target anti-crawler model, and the corresponding target response measures are executed based on the comparison result to perform anti-crawler processing, and the processing results are saved to a local database. In this way, through the above process of this application, the user behavior feature data is compared using the results output by the preset target anti-crawler model and the preset judgment rules to realize intelligent identification of malicious crawlers, which can effectively reduce the misjudgment rate and improve the security of the website, thereby improving the access experience of normal users. At the same time, based on the comparison results, corresponding target response measures are executed to perform anti-crawler processing, thereby realizing intelligent identification of malicious crawlers and automatically executing corresponding anti-crawler measures. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0044] Figure 1 A flow chart of an anti-crawler method disclosed in this application;

[0045] Figure 2 A schematic diagram of the flow of an anti-crawler method disclosed in this application;

[0046] Figure 3 A specific anti-crawler method flow chart disclosed in this application;

[0047] Figure 4 This is a schematic diagram of the structure of an anti-crawler device disclosed in this application;

[0048] Figure 5 This is a structural diagram of an electronic device disclosed in this application. DETAILED DESCRIPTION

[0049] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0050] Currently, anti-crawler technology is usually implemented through methods such as verification code verification and real-time traffic monitoring. However, these existing technologies have low accuracy in identifying malicious crawlers, resulting in high false positive and missed positive rates for malicious crawlers. At the same time, methods that require user intervention, such as verification code verification, will affect the access experience of normal users and cannot effectively implement anti-crawler technology.

[0051] In order to overcome the above technical problems, the present application provides an anti-crawler method to intelligently identify malicious crawlers and automatically execute corresponding anti-crawler measures.

[0052] See also Figure 1 As shown, an embodiment of the present invention discloses an anti-crawler method, which is applied to an anti-crawler system, including:

[0053] Step S11 : performing real-time monitoring on the monitored client to obtain real-time user access data and real-time user behavior monitoring data, and extracting user behavior feature data based on the real-time user access data and the real-time user behavior monitoring data.

[0054] In this embodiment, the client to be monitored is monitored in real time to obtain real-time user access data and real-time user behavior monitoring data from the client to be monitored, and the obtained real-time user access data and real-time user behavior monitoring data are used to extract user behavior feature data. Specifically, this embodiment can collect user access data and use front-end scripts to monitor user behavior, such as user mouse movement, clicks, scrolling, etc., while monitoring the client to be monitored in real time, to obtain real-time user access data and real-time user behavior monitoring data, and use the obtained data to extract user behavior features. Among them, the access data includes but is not limited to request frequency, request path, user agent, access time, etc.; the client to be monitored is a variety of clients that need to be monitored in real time for crawler detection and anti-crawler processing based on the detection results, such as game application clients, browser clients integrated with game programs, instant messaging clients, etc.

[0055] It is understandable that, in addition to monitoring the client, this embodiment can also monitor the terminal and the server for anti-crawler processing. The terminal includes but is not limited to smart phones, tablet computers, laptops, touch screens, game consoles, personal computers (i.e., Personal Computers, PCs), personal digital assistants (i.e., Personal Digital Assistants, PDAs), etc.; the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers.

[0056] In order to realize the function of intelligently identifying malicious crawlers, this embodiment needs to train a classification model for crawler identification before real-time monitoring of the monitored client. The processing process is as follows: collect user access data from the monitored client, and monitor user behavior to obtain user behavior monitoring data; extract first behavior feature data from the user access data, and use the user behavior monitoring data to construct second behavior feature data; preprocess the first behavior feature data and the second behavior feature data to construct a target feature set based on the target feature data obtained after preprocessing; train the original model based on the target feature set to use the trained model as the preset target anti-crawler model. Specifically, the user access data is continuously collected from the client to be monitored, and the relevant information of each user request is recorded, such as the Internet Protocol (IP) address, request time, request path, request method, user agent, etc. At the same time, the front-end script is integrated to monitor user behavior, such as mouse movement trajectory, page dwell time, scrolling behavior, etc., to obtain the user behavior monitoring data, and after ensuring that the collected data format is unified, the user access data and the user behavior monitoring data are saved to the local database for subsequent analysis of the user access data and the user behavior monitoring data. Subsequently, the first behavior feature data is extracted from the user access data in the local database, and the second behavior feature data is constructed using the user behavior monitoring data. The first behavior feature data and the second behavior feature data are preprocessed to eliminate dimensional differences and ensure the effect of model training, and a target feature set is constructed based on the target feature data obtained after preprocessing. Among them, the first behavior feature data includes but is not limited to user access frequency, user request interval, page access depth, etc.; the use of the user behavior monitoring data to construct the second behavior feature data can be by analyzing the user's regular access path to obtain the access pattern, or by calculating the number of user interactions on the page to obtain the interaction characteristics, or based on abnormally high request frequency or user behavior monitoring data that accesses a large number of pages in a short period of time to determine abnormal behavior characteristics, etc. Finally, the original model is trained using the target feature set and the obtained trained model is used as the preset target anti-crawler model. It can be understood that this embodiment can regularly compare the actual crawler behavior features with the feature data in the target feature set to optimize the extraction strategy of the first behavior feature data and the construction strategy of the second behavior feature data according to the comparison results obtained, so as to improve the adaptability of the model.

[0057] It should be noted that in order to avoid the situation that the performance of the target anti-crawler model finally trained is low and the malicious crawler cannot be effectively identified due to the presence of data that affects the data quality, such as duplicate data, missing data, abnormal data, etc. in the first behavior feature data and the second behavior feature data, and to ensure the stability of the model during training, it is necessary to pre-process the first behavior feature data and the second behavior feature data, and the processing flow is as follows: remove the duplicate data, missing data and abnormal data in the first behavior feature data and the second behavior feature data to obtain the removed feature data; perform one-hot encoding on the classification features in the removed feature data, and normalize or standardize the numerical features to obtain the target feature data. That is, in this embodiment, the duplicate data, missing data and abnormal data in the first behavior feature data and the second behavior feature data are removed to ensure the quality of the data, and at the same time, the classification features in the removed feature data are uniquely encoded, and the numerical features in the removed feature data are normalized or standardized to ensure the stability of the model during training, and the performance of the target anti-crawler model finally trained is improved to effectively identify malicious crawlers.

[0058] In this embodiment, the processing flow of training the original model based on the target feature set to use the obtained trained model as the preset target anti-crawler model is as follows: select a suitable machine learning algorithm according to data characteristics and business needs, and divide the target feature set to obtain a divided training data set, a verification data set and a test data set; train the original model based on the training data set and preset hyperparameters to improve the performance of the model, and use the verification data set to monitor the performance of the model to prevent overfitting, and finally obtain the trained model; evaluate the generalization ability and stability of the model based on the test data set, preset evaluation indicators and K-fold cross-validation method to ensure the reliability of the results; optimize the preset hyperparameters using a grid search or random search method based on the obtained evaluation results, and use a feature selection algorithm to evaluate the importance of the target feature set to determine the most effective target features for identifying malicious crawlers; train the trained model using the optimized preset hyperparameters and the target features to obtain the target anti-crawler model. Among them, the machine learning algorithm includes but is not limited to support vector machine, random forest, gradient boosting tree, deep learning, etc.; the training data set, the verification data set and the test data set can be obtained by dividing the target feature set according to the ratio of 14:3:3; the preset evaluation indicators include but are not limited to accuracy, precision, recall, F1 score (F1-score), etc.; the feature selection algorithm includes but is not limited to chi-square test algorithm, random forest feature importance algorithm, etc. In this way, this embodiment first trains a classification model for crawler identification based on user access data and user behavior monitoring data collected from the client to be monitored to realize the function of intelligently identifying malicious crawlers, and then performs real-time monitoring on the client to be monitored to extract user behavior feature data based on the real-time user access data and real-time user behavior monitoring data obtained through monitoring, so as to obtain user behavior feature data more comprehensively, which is convenient for improving the accuracy of subsequent intelligent identification of malicious crawlers.

[0059] Step S12: input the user behavior feature data into a preset target anti-crawler model, and compare the user behavior feature data according to preset judgment rules and the output result of the target anti-crawler model, and then determine the target response measures corresponding to the comparison result based on the comparison result.

[0060] In this embodiment, the user behavior characteristic data is input into a preset target anti-crawler model to obtain the output result of the target anti-crawler model, and the user behavior characteristic data is compared according to the preset judgment rules and the output result, and the type of the user behavior characteristic data is determined based on the obtained comparison result, so as to determine the corresponding target response measures according to the type. Among them, the target response measures include but are not limited to allowing the user to continue to access, triggering a verification code, reducing the access frequency, limiting the access time, temporarily or permanently banning the IP, etc.; the types of the user behavior characteristic data include but are not limited to suspicious user behavior characteristic data, normal user behavior characteristic data, malicious crawler behavior characteristic data, etc.

[0061] It is understandable that this embodiment can periodically analyze the historical processing results stored in the local database to obtain the malicious behavior trend characteristics of the malicious crawler and determine whether there is a new malicious crawler attack mode; if there is a new malicious crawler attack mode, the preset judgment rule and the response measures currently stored in the local database are updated based on the malicious behavior trend characteristics and the new malicious crawler attack mode; if there is no new malicious crawler attack mode, the preset judgment rule and the response measures currently stored in the local database are updated based on the malicious behavior trend characteristics. Among them, the malicious behavior trend characteristics refer to the continuous and regular change characteristics of the malicious crawler activities over a period of time. That is, the historical processing results recorded in the local database are analyzed based on the preset frequency to obtain the malicious behavior trend characteristics of the malicious crawler, and the preset judgment rules and the response measures currently stored in the local database are updated based on the malicious behavior trend characteristics, so that the anti-crawler strategy can be adjusted more effectively, and at the same time, it is determined whether there is a new malicious crawler attack mode in the historical processing results. If there is a new malicious crawler attack mode, it means that if the malicious crawler corresponding to the new malicious crawler attack mode appears again, based on the current preset judgment rules and the current response measures stored in the local database, it may not be possible to effectively identify the malicious crawler and perform anti-crawler processing. The preset judgment rules and the response measures currently stored in the local database can be updated based on the malicious behavior trend characteristics and the new malicious crawler attack mode. By determining the malicious behavior trend characteristics and whether there is a new malicious crawler attack mode, the anti-crawler strategy can be adjusted more effectively. At the same time, by continuously updating the preset judgment rules and the response measures, the adaptability and protection capability of the anti-crawler system are improved, and the missed judgment rate of the anti-crawler system is reduced. In this way, this embodiment inputs the user behavior feature data into the preset target anti-crawler model to intelligently identify the user behavior feature data, and then determines the corresponding target response measures based on the output results of the target anti-crawler model to perform anti-crawler processing. At the same time, by analyzing the recorded historical processing results, the preset judgment rules and saved response measures are continuously updated, thereby improving the adaptability of the anti-crawler system, being able to more flexibly identify user behavior feature data, and reducing the possibility of misjudgment.

[0062] Step S13: execute the target response measures to perform anti-crawler processing, and save the processing results to the local database.

[0063] In this embodiment, anti-crawler processing is performed by executing the determined target response measures, and the processing results are saved in the local database. Specifically, the processing flow of executing the target response measures to perform anti-crawler processing is as follows: executing the target response measures to perform user ban or access restriction and request rate restriction on the suspicious user corresponding to the suspicious user behavior feature data; or, performing an alarm operation on the suspicious user, and after receiving the audit result corresponding to the alarm operation, training the target anti-crawler model based on the audit result. That is, if the comparison result characterizes that the user behavior feature data is suspicious user behavior feature data, then the suspicious user corresponding to the suspicious user behavior feature data is user banned or access restricted and request rate restricted, or, generating alarm information for the suspicious user, and sending the alarm information to the terminal where the system-related management personnel are located, so that the system-related management personnel can review the suspicious user and mark the suspicious user behavior feature data, and feedback the audit results and the marked data to the anti-crawler system, and finally update the target anti-crawler model based on the received audit results and the marked data to form a closed-loop feedback mechanism. In addition, the suspicious user may be required to undergo a Completely Automated Public Turing Test to Tell Humans Apart (CAPTCHA) verification to confirm the identity of the suspicious user. It should be noted that the above-mentioned alarm operation on the suspicious user is performed on the premise that the suspicious degree of the suspicious user exceeds a preset threshold. Specifically, if the suspicious user is a highly suspicious user, an alarm operation is performed on the suspicious user, and the suspicious user is determined to be a malicious crawler based on the received audit results, and the target anti-crawler model is updated and trained. Among them, the basis for determining whether the suspicious user is a highly suspicious user may be to determine that the difference between the result output by the target anti-crawler model and the preset abnormal threshold exceeds a preset quantity threshold, or the target anti-crawler model cannot identify the suspicious user behavior feature data corresponding to the suspicious user.

[0064] It should be noted that in order to improve the adaptability and reliability of the anti-crawler method, this embodiment can continuously optimize and update the anti-crawler method after saving the processing results to the local database. The processing flow is as follows: based on the preset update frequency, the target feature set is updated by newly added feature data to obtain an updated feature set; wherein the newly added feature data is behavioral feature data obtained by feature extraction of newly added user behavior monitoring data; the target anti-crawler model is evaluated for performance impact through the updated feature set to determine whether there is feature data to be optimized in the updated feature set according to the evaluation results; the performance impact evaluation is to evaluate the performance impact test of the target anti-crawler model through the feature data in the target feature set; if there is feature data to be optimized in the updated feature set, the feature data to be optimized is removed from the updated feature set to obtain the target updated feature set; the target anti-crawler model is updated and trained through the target updated feature set to use the trained anti-crawler model as the current target anti-crawler model. That is, this embodiment regularly uses the newly collected additional feature data to update the target feature set to train and update the target anti-crawler model to adjust its decision boundary, ensure that the target anti-crawler model always maintains a high accuracy, improve its recognition ability, and regularly evaluate the impact of features on model performance to identify the feature data to be optimized in the target feature set, that is, features that are invalid in the new environment or have reduced importance in identifying crawlers, and remove the feature data to be optimized from the target feature set to enhance the adaptability of the model. Figure 2 The figure shows a schematic diagram of the overall anti-crawler method flow.

[0065] Furthermore, in order to ensure that the target anti-crawler model can quickly adapt to new crawler behaviors without occupying a large amount of computing resources, this embodiment can use an incremental learning algorithm to update the existing target anti-crawler model when new feature data arrives, avoiding training from scratch and occupying unnecessary computing resources. In this way, this embodiment executes anti-crawler processing based on the target response measures determined by the comparison results, and saves the processing results to the local database, so that the behavioral trend characteristics of malicious crawlers can be understood in a timely manner based on the saved processing results, and then the anti-crawler strategy can be dynamically adjusted according to the changes in crawler technology, so as to improve the sensitivity of the target anti-crawler model to the newly added malicious crawler attack mode.

[0066] As can be seen from the above, the embodiment of the present application performs real-time monitoring of the monitored client to obtain relevant real-time user access data and real-time user behavior monitoring data, and extracts user behavior feature data based on the acquired real-time user access data and the real-time user behavior monitoring data, and inputs the extracted user behavior feature data into a preset target anti-crawler model to compare the user behavior feature data based on preset judgment rules and the output result of the target anti-crawler model, and executes corresponding target response measures based on the comparison result to perform anti-crawler processing, and saves the processing result to a local database. In this way, through the above process of the embodiment of the present application, on the one hand, the monitored client is monitored in real time to extract user behavior feature data based on the real-time user access data and real-time user behavior monitoring data obtained through monitoring, so as to obtain user behavior feature data more comprehensively, which is convenient for improving the accuracy of subsequent intelligent identification of malicious crawlers; on the other hand, the user behavior feature data is compared with the results output by the preset target anti-crawler model and the preset judgment rules to realize intelligent identification of malicious crawlers, which can effectively reduce the misjudgment rate and improve the security of the website, thereby improving the access experience of normal users; on the other hand, based on the comparison results, corresponding target response measures are executed to perform anti-crawler processing, and the processing results are saved in the local database, so as to continue to understand the behavior trend characteristics of malicious crawlers in a timely manner based on the saved processing results, so as to dynamically adjust the anti-crawler strategy according to the changes in crawler technology, and improve the sensitivity of the target anti-crawler model to the newly added malicious crawler attack mode, thereby realizing intelligent identification of malicious crawlers and automatically executing corresponding anti-crawler measures.

[0067] Based on the above embodiments, it can be seen that through the method of the present application, after the user behavior feature data is input into the preset target anti-crawler model, the user behavior feature data can be compared according to the output result of the target anti-crawler model and the preset judgment rule, so as to determine the corresponding target response measures according to the obtained comparison results. For this reason, this embodiment describes in detail how to compare the user behavior feature data and determine the corresponding target response measures. Figure 3 As shown, an embodiment of the present invention discloses an anti-crawler method, which is applied to an anti-crawler system, including:

[0068] Step S21 : performing real-time monitoring on the client to be monitored to obtain real-time user access data and real-time user behavior monitoring data, and extracting user behavior feature data based on the real-time user access data and the real-time user behavior monitoring data.

[0069] Step S22: input the user behavior feature data into a preset target anti-crawler model to obtain a predicted abnormal probability distribution corresponding to the user behavior feature data output by the target anti-crawler model.

[0070] In this embodiment, the output result of the target anti-crawler model is obtained by inputting the user behavior feature data into the preset target anti-crawler model, that is, the predicted abnormal probability distribution corresponding to the user behavior feature data. It can be understood that the predicted abnormal probability distribution represents the possibility that the user behavior feature data is identified as abnormal behavior feature data. The predicted abnormal probability distribution combined with the preset judgment rules helps to judge whether the user behavior feature data is a malicious crawler, and then perform anti-crawler processing to improve network security.

[0071] Step S23: Compare the predicted abnormal probability distribution with the preset abnormal threshold to obtain a corresponding comparison result.

[0072] In this embodiment, the predicted abnormal probability distribution corresponding to the user behavior feature data output by the target anti-crawler model is compared with the preset abnormal threshold, so as to obtain a comparison result of whether there is a target user behavior feature data higher than the preset abnormal threshold in the predicted abnormal probability distribution. Among them, the preset abnormal threshold can be set in advance by the system-related administrator according to business needs, security policies, model performance and other factors, and is usually set to one or more specific values ​​to distinguish normal user behavior feature data from suspicious user behavior feature data. For example, if the request frequency in the user behavior feature data exceeds the preset abnormal threshold, or the number of user page visits exceeds the preset abnormal threshold in a short time, the user behavior feature data can be determined as suspicious user behavior feature data. Specifically, when comparing the predicted abnormal probability distribution with the preset abnormal threshold, this embodiment can traverse each probability value in the predicted abnormal probability distribution, and compare each probability value with the preset abnormal threshold. If there is a target probability value higher than the preset abnormal threshold, the user behavior feature data corresponding to the target probability value can be determined as suspicious user behavior feature data.

[0073] It is understandable that if the preset abnormal threshold is set too high, some truly suspicious user behavior feature data may be missed, and the security of the website is difficult to guarantee; and if the preset abnormal threshold is set too low, it may result in a high misjudgment rate, misjudging normal user behavior feature data as suspicious user behavior feature data. Therefore, the setting of the preset abnormal threshold is very important. At the same time, as crawler technology is constantly improving and new types of crawlers are also increasing, in order to ensure the accuracy of the anti-crawler system, this embodiment can regularly adjust and optimize the preset abnormal threshold based on user feedback, business needs and other factors. In this way, this embodiment compares the predicted abnormal probability distribution with the preset abnormal threshold to obtain a corresponding comparison result, so as to determine the user behavior feature data based on the comparison result, and execute the corresponding target response measures. At the same time, the preset abnormal threshold is set to compare with the predicted abnormal probability distribution output by the target anti-crawler model, which can realize intelligent identification of malicious crawlers and ensure the security of the website.

[0074] Step S24: If the comparison result indicates that there is no target user behavior characteristic data higher than the preset abnormal threshold in the predicted abnormal probability distribution, then the user behavior characteristic data is determined to be normal user behavior characteristic data.

[0075] In this embodiment, if the comparison result represents that there is no target user behavior characteristic data higher than the preset abnormal threshold in the predicted abnormal probability distribution, it means that there is no abnormal behavior in the user behavior characteristic data, and the user behavior characteristic data can be judged as normal user behavior characteristic data, and normal users corresponding to the normal user behavior characteristic data are allowed to continue to access, so as to improve the user's access experience by timely judging the user behavior characteristic data and executing corresponding response measures.

[0076] Step S25: If the comparison result indicates that there is target user behavior feature data higher than the preset abnormal threshold in the predicted abnormal probability distribution, the target user behavior feature data is used as suspicious user behavior feature data, and a target response measure corresponding to the suspicious user behavior feature data is determined in a local database.

[0077] In this embodiment, if the comparison result indicates that there is target user behavior characteristic data higher than the preset abnormal threshold in the predicted abnormal probability distribution, it means that there is abnormal behavior in the target user behavior characteristic data, and the target user behavior characteristic data can be used as suspicious user behavior characteristic data, and the target response measures corresponding to the suspicious user behavior characteristic data can be determined in the local database. For example, the suspicious user corresponding to the suspicious user behavior characteristic data can be required to undergo CAPTCHA verification to confirm its identity; for the IP corresponding to the high-frequency access suspicious user behavior characteristic data, temporary access restrictions can be implemented to reduce its request rate; for the suspicious user behavior characteristic data confirmed as a malicious crawler, the IP of the suspicious user behavior characteristic data can be added to the blacklist to prevent its subsequent access behavior. In this way, after the target user behavior characteristic data is used as suspicious user behavior characteristic data, the embodiment can determine the corresponding target response measures from the local database based on the analysis of the suspicious user behavior characteristic data, thereby improving the effectiveness of anti-crawler processing.

[0078] Step S26: execute the target response measure to perform anti-crawler processing, and save the processing result to the local database.

[0079] The specific implementation process of steps S21 and S26 may refer to the contents of the aforementioned embodiments and will not be described in detail here.

[0080] As can be seen from the above, the embodiment of the present application performs real-time monitoring of the monitored client to obtain relevant real-time user access data and real-time user behavior monitoring data, and extracts user behavior feature data based on the acquired real-time user access data and the real-time user behavior monitoring data, and inputs the extracted user behavior feature data into a preset target anti-crawler model to obtain the predicted abnormal probability distribution corresponding to the user behavior feature data output by the target anti-crawler model, and compares the predicted abnormal probability distribution with the preset abnormal threshold, judges the user behavior feature data based on the comparison result, executes corresponding target response measures to perform anti-crawler processing, and saves the processing results to a local database. In this way, through the above process of the embodiment of the present application, on the one hand, by inputting the user behavior feature data into the preset target anti-crawler model to obtain the output predicted abnormal probability distribution, it is helpful to determine whether the user behavior feature data is a malicious crawler, and then perform anti-crawler processing to improve network security; on the other hand, by comparing the predicted abnormal probability distribution with the preset abnormal threshold, it is possible to intelligently identify malicious crawlers and ensure the security of the website; on the other hand, after judging the user behavior feature data based on the comparison results, the corresponding target response measures are determined in a targeted manner, which can improve the effectiveness of anti-crawler processing and the user's access experience, thereby realizing intelligent identification of malicious crawlers and automatically executing corresponding anti-crawler measures.

[0081] Accordingly, see Figure 4 As shown, the embodiment of the present application also provides an anti-crawler device, which is applied to the anti-crawler system, including:

[0082] The feature extraction module 11 is used to perform real-time monitoring on the monitored client to obtain real-time user access data and real-time user behavior monitoring data, and extract user behavior feature data based on the real-time user access data and the real-time user behavior monitoring data;

[0083] A measure determination module 12 is used to input the user behavior feature data into a preset target anti-crawler model, and compare the user behavior feature data according to a preset determination rule and an output result of the target anti-crawler model, and then determine a target response measure corresponding to the comparison result based on the obtained comparison result;

[0084] The result saving module 13 is used to execute the target response measures to perform anti-crawler processing and save the processing results to a local database.

[0085] As can be seen from the above, the embodiment of the present application performs real-time monitoring of the monitored client to obtain relevant real-time user access data and real-time user behavior monitoring data, and extracts user behavior feature data based on the obtained real-time user access data and the real-time user behavior monitoring data, and inputs the extracted user behavior feature data into the preset target anti-crawler model, so as to compare the user behavior feature data based on the preset judgment rules and the output results of the target anti-crawler model, and execute corresponding target response measures based on the comparison results to perform anti-crawler processing, and save the processing results to the local database. In this way, through the above process of the embodiment of the present application, the user behavior feature data is compared using the results of the preset target anti-crawler model output and the preset judgment rules to realize intelligent identification of malicious crawlers, which can effectively reduce the misjudgment rate, improve the security of the website, and thus improve the access experience of normal users. At the same time, based on the comparison results, the corresponding target response measures are executed to perform anti-crawler processing, thereby realizing intelligent identification of malicious crawlers and automatically executing corresponding anti-crawler measures.

[0086] In some specific implementations, the feature extraction module 11 may further include:

[0087] A behavior monitoring submodule, used to collect user access data from the client to be monitored, and monitor user behavior to obtain user behavior monitoring data;

[0088] A data construction submodule, used to extract first behavior characteristic data from the user access data, and construct second behavior characteristic data using the user behavior monitoring data;

[0089] A data preprocessing submodule, used for preprocessing the first behavior feature data and the second behavior feature data to construct a target feature set based on the target feature data obtained after the preprocessing;

[0090] The model training submodule is used to train the original model based on the target feature set to use the obtained trained model as a preset target anti-crawler model.

[0091] In some specific implementations, the data preprocessing submodule may specifically include:

[0092] A data elimination unit, used to eliminate duplicate data, missing data and abnormal data in the first behavior characteristic data and the second behavior characteristic data to obtain eliminated characteristic data;

[0093] The data processing unit is used to perform one-hot encoding on the categorical features in the eliminated feature data and normalize or standardize the numerical features to obtain target feature data.

[0094] In some specific implementations, the anti-crawler device may further include:

[0095] A feature set updating unit, configured to update the target feature set by adding feature data based on a preset update frequency to obtain an updated feature set; wherein the newly added feature data is behavior feature data obtained by extracting features from newly added user behavior monitoring data;

[0096] A model evaluation unit, configured to perform a performance impact evaluation on the target anti-crawler model through the updated feature set, so as to determine whether there is feature data to be optimized in the updated feature set according to the evaluation result; the performance impact evaluation is an evaluation of a performance impact test on the target anti-crawler model through the feature data in the target feature set;

[0097] a data removing unit, configured to remove the feature data to be optimized from the updated feature set if there is feature data to be optimized in the updated feature set, so as to obtain a target updated feature set;

[0098] A model updating unit is used to update and train the target anti-crawler model using the target updated feature set, so as to use the obtained trained anti-crawler model as the current target anti-crawler model.

[0099] In some specific implementations, the measure determination module 12 may specifically include:

[0100] A data input unit, used to input the user behavior feature data into a preset target anti-crawler model to obtain a predicted abnormal probability distribution corresponding to the user behavior feature data output by the target anti-crawler model;

[0101] An abnormality comparison unit, used to compare the predicted abnormality probability distribution with a preset abnormality threshold to obtain a corresponding comparison result;

[0102] a condition determination unit, configured to determine that the user behavior characteristic data is normal user behavior characteristic data if the comparison result indicates that there is no target user behavior characteristic data higher than the preset abnormal threshold in the predicted abnormal probability distribution;

[0103] A measure determination unit is used to, if the comparison result indicates that there is target user behavior characteristic data in the predicted abnormal probability distribution that is higher than the preset abnormal threshold, use the target user behavior characteristic data as suspicious user behavior characteristic data, and determine a target response measure corresponding to the suspicious user behavior characteristic data in the local database.

[0104] In some specific implementations, the result saving module 13 may specifically include:

[0105] A measure execution unit, configured to execute the target response measure to ban or restrict access and limit request rate of the suspicious user corresponding to the suspicious user behavior feature data;

[0106] Or, a model training unit is used to perform an alarm operation on the suspicious user, and after receiving an audit result corresponding to the alarm operation, train the target anti-crawler model based on the audit result.

[0107] In some specific embodiments, the anti-crawler device may further include:

[0108] A result analysis unit is used to analyze the historical processing results stored in the local database to obtain the malicious behavior trend characteristics of the malicious crawler and determine whether there is a new malicious crawler attack mode;

[0109] A first measure updating unit, configured to update the preset determination rule and the response measures currently stored in the local database based on the malicious behavior trend characteristics and the newly added malicious crawler attack pattern if the newly added malicious crawler attack pattern exists;

[0110] The second measure updating unit is used to update the preset judgment rule and the response measures currently stored in the local database based on the malicious behavior trend characteristics if the newly added malicious crawler attack mode does not exist.

[0111] Furthermore, the present application also discloses an electronic device. Figure 5 It is a structural diagram of an electronic device 20 shown according to an exemplary embodiment, and the content in the figure cannot be regarded as any limitation on the scope of use of this application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input and output interface 25 and a communication bus 26. Among them, the memory 22 is used to store a computer program, and the computer program is loaded and executed by the processor 21 to implement the relevant steps in the anti-crawler method disclosed in any of the aforementioned embodiments. In addition, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0112] In this embodiment, the power supply 23 is used to provide working voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and the external device, and the communication protocol it follows is any communication protocol that can be applied to the technical solution of the present application, and is not specifically limited here; the input and output interface 25 is used to obtain external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs and is not specifically limited here.

[0113] In addition, the memory 22 as a carrier for resource storage may be a read-only memory, a random access memory, a disk or an optical disk, etc. The resources stored thereon may include an operating system 221, a computer program 222, etc., and the storage method may be temporary storage or permanent storage.

[0114] The operating system 221 is used to manage and control the hardware devices and computer program 222 on the electronic device 20, which can be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program that can be used to complete the anti-crawler method performed by the electronic device 20 disclosed in any of the aforementioned embodiments, the computer program 222 can further include a computer program that can be used to complete other specific tasks.

[0115] Furthermore, the present application also discloses a computer-readable storage medium for storing a computer program; wherein the computer program, when executed by a processor, implements the anti-crawler method disclosed above. The specific steps of the method can refer to the corresponding contents disclosed in the above embodiments, and will not be repeated here.

[0116] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0117] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0118] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0119] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0120] The technical solution provided by the present application is introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for general technicians in this field, according to the idea of ​​the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. An anti-crawler method, characterized in that: Applied to anti-crawler systems, including: Performing real-time monitoring on the monitored client to obtain real-time user access data and real-time user behavior monitoring data, and extracting user behavior feature data based on the real-time user access data and the real-time user behavior monitoring data; Inputting the user behavior feature data into a preset target anti-crawler model, and comparing the user behavior feature data according to a preset determination rule and an output result of the target anti-crawler model, and then determining a target response measure corresponding to the comparison result based on the obtained comparison result; The target response measures are executed to perform anti-crawler processing, and the processing results are saved to a local database.

2. The anti-crawler method according to claim 1, characterized in that: Before performing real-time monitoring on the client to be monitored to obtain real-time user access data and real-time user behavior monitoring data, the method further includes: Collecting user access data from the client to be monitored, and monitoring user behavior to obtain user behavior monitoring data; Extracting first behavior characteristic data from the user access data, and constructing second behavior characteristic data using the user behavior monitoring data; Preprocessing the first behavior feature data and the second behavior feature data to construct a target feature set based on the target feature data obtained after the preprocessing; The original model is trained based on the target feature set to use the obtained trained model as a preset target anti-crawler model.

3. The anti-crawler method according to claim 2, characterized in that: The preprocessing of the first behavior characteristic data and the second behavior characteristic data includes: Eliminate duplicate data, missing data, and abnormal data in the first behavior characteristic data and the second behavior characteristic data to obtain eliminated characteristic data; The categorical features in the removed feature data are one-hot encoded, and the numerical features are normalized or standardized to obtain target feature data.

4. The anti-crawler method according to claim 3, characterized in that: After executing the target response measures to perform anti-crawler processing and saving the processing results to the local database, it also includes: Based on a preset update frequency, the target feature set is updated by adding feature data to obtain an updated feature set; wherein the newly added feature data is behavior feature data obtained by extracting features from newly added user behavior monitoring data; A performance impact assessment is performed on the target anti-crawler model through the updated feature set to determine whether there is feature data to be optimized in the updated feature set according to the assessment result; the performance impact assessment is an assessment of a performance impact test performed on the target anti-crawler model through the feature data in the target feature set; If there is feature data to be optimized in the updated feature set, the feature data to be optimized is removed from the updated feature set to obtain a target updated feature set; The target anti-crawler model is updated and trained using the target updated feature set, so that the obtained trained anti-crawler model is used as the current target anti-crawler model.

5. The anti-crawler method according to claim 1, characterized in that: The step of inputting the user behavior feature data into a preset target anti-crawler model, comparing the user behavior feature data according to a preset determination rule and an output result of the target anti-crawler model, and then determining a target response measure corresponding to the comparison result based on the obtained comparison result includes: Inputting the user behavior feature data into a preset target anti-crawler model to obtain a predicted abnormal probability distribution corresponding to the user behavior feature data output by the target anti-crawler model; Comparing the predicted abnormal probability distribution with a preset abnormal threshold to obtain a corresponding comparison result; If the comparison result indicates that there is no target user behavior feature data higher than the preset abnormality threshold in the predicted abnormal probability distribution, then the user behavior feature data is determined to be normal user behavior feature data; If the comparison result indicates that there is target user behavior feature data higher than the preset abnormality threshold in the predicted abnormal probability distribution, the target user behavior feature data is used as suspicious user behavior feature data, and a target response measure corresponding to the suspicious user behavior feature data is determined in the local database.

6. The anti-crawler method according to claim 5, characterized in that: The executing the target response measure to perform anti-crawler processing includes: Executing the target response measures to ban or restrict access and limit request rate for the suspicious user corresponding to the suspicious user behavior feature data; Or, perform an alarm operation on the suspicious user, and after receiving the audit result corresponding to the alarm operation, train the target anti-crawler model based on the audit result.

7. The anti-crawler method according to any one of claims 1 to 6, characterized in that: Also includes: Analyze the historical processing results saved in the local database to obtain the malicious behavior trend characteristics of malicious crawlers and determine whether there are new malicious crawler attack patterns; If the newly added malicious crawler attack pattern exists, the preset judgment rule and the response measures currently stored in the local database are updated based on the malicious behavior trend characteristics and the newly added malicious crawler attack pattern; If the newly added malicious crawler attack pattern does not exist, the preset judgment rules and the response measures currently stored in the local database are updated based on the malicious behavior trend characteristics.

8. An anti-crawler device, characterized in that: Applied to anti-crawler systems, including: A feature extraction module is used to perform real-time monitoring on the monitored client to obtain real-time user access data and real-time user behavior monitoring data, and extract user behavior feature data based on the real-time user access data and the real-time user behavior monitoring data; A measure determination module, used for inputting the user behavior feature data into a preset target anti-crawler model, and comparing the user behavior feature data according to a preset determination rule and an output result of the target anti-crawler model, and then determining a target response measure corresponding to the comparison result based on the comparison result; The result saving module is used to execute the target response measures to perform anti-crawler processing and save the processing results to a local database.

9. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, used to execute the computer program to implement the anti-crawler method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: Used to store computer programs; wherein, when the computer program is executed by a processor, the anti-crawler method according to any one of claims 1 to 7 is implemented.

Images