Anti-crawler monitoring and processing method, device, equipment and medium

Abstract

The present application provides an anti-crawler monitoring and processing method, device, equipment and medium, which relates to the field of network security technology, including: obtaining access behavior parameters corresponding to the current access behavior based on the access data and access log of the current access behavior obtained in real time; constructing an access behavior feature matrix based on the access log updated based on the access data, and obtaining the access feature parameters corresponding to the current access behavior from the access behavior feature matrix; obtaining a preset crawler recognition model obtained by model training based on sample access behavior parameters and sample access feature parameters corresponding to multiple historical access behaviors and their sample crawler results, and obtaining the crawler recognition result corresponding to the current access behavior; and performing crawler confrontation when the current access behavior corresponding to the crawler recognition result is a malicious crawler behavior. The present application improves the efficiency and comprehensiveness of anti-crawler defense, ensures normal response to legitimate user requests and effectively protects sensitive information and encrypted data.

Description

Technical Field

[0001] The present application relates to the field of computer network security technology, and in particular to an anti-crawler monitoring and processing method, device, equipment and medium. Background Art

[0002] Web crawlers, also known as web spiders, are programs or scripts that automatically crawl the World Wide Web according to certain rules. However, malicious crawlers can exploit website vulnerabilities to cause data leakage and privacy violations, server pressure and performance degradation, and website structure and content tampering. Therefore, it is necessary to identify and defend against malicious crawler behaviors.

[0003] In the prior art, the defense method against crawler behavior usually includes measures such as creating a proxy pool, setting anti-crawler strategies, and performing verification code identification and verification. For example, the invention patent with application number CN202311216608.3 involves an efficient anti-crawler method that integrates detection and countermeasures. First, a proxy pool is created, and the most available proxy is screened out through the detection module; then the configuration information of this task is set, and the anti-crawler detection module uses Selenium technology to detect the target website according to the proxy and configuration information, and writes the detection results into the configuration file; then the crawler thread takes the task out of the initialization queue and carries the parameters to automatically log in to the target website. If there is a verification code on the web page, take a screenshot of the tag where the verification code is located and save it, and then call the OCR interface for identification. After successful login, obtain the JWT token and parse it to detect whether it is valid. If it is invalid, you need to log in again. If it is valid, you can set a random speed limit, and finally start the crawler collection task. For another example, the invention patent with application number CN201610127493.4 provides a server, an anti-crawler system and an anti-crawler verification method. The server is connected to a site in communication, and the server stores a published anti-crawler strategy for countering crawler behavior against the site. The anti-crawler verification method includes: S1, the site sends a message to the server to obtain the anti-crawler strategy to the user; S2, the server transmits the anti-crawler strategy to the user after receiving the control instruction sent by the user; S3, the site receives the decryption result for the anti-crawler strategy sent by the user, and sends the decryption result to the server; S4, the server verifies the decryption result and sends the verification result to the site, and the verification result is information that the decryption result is correct or information that the decryption result is wrong.

[0004] However, the existing anti-crawler measures still have certain limitations when facing complex and changeable crawler behaviors; especially in scenarios where sensitive information or encrypted data needs to be protected, the existing defense measures cannot fully and effectively identify and prevent malicious crawler behaviors. Therefore, there is an urgent need for a more efficient and comprehensive anti-crawler method that can ensure that server resources can normally respond to legitimate user requests and effectively protect sensitive information and encrypted data. Summary of the invention

[0005] The purpose of this application is to address the above problems and provide an anti-crawler monitoring and processing method, device, equipment and medium to improve the efficiency and comprehensiveness of anti-crawler defense, ensure that server resources respond normally to the requests of legitimate users, and effectively protect sensitive information and encrypted data.

[0006] In a first aspect, the present application provides an anti-crawler monitoring and processing method, comprising:

[0007] Obtain access data of current access behavior in real time;

[0008] Obtaining the access log, and obtaining the access behavior parameters corresponding to the current access behavior according to the access data and the access log; the access behavior parameters include access type, access interval, access frequency, and behavior pattern;

[0009] The access log is updated based on the access data, and an access behavior feature matrix is ​​constructed according to the updated access log, and access feature parameters corresponding to the current access behavior are obtained from the access behavior feature matrix; the access feature parameters include crawler risk type, access frequency standard deviation, access frequency mean, and access time distribution sequence;

[0010] Input the access behavior parameter and the access feature parameter into a preset crawler recognition model to obtain a crawler recognition result corresponding to the current access behavior output by the preset crawler recognition model; the preset crawler recognition model is obtained by model training based on sample access behavior parameters and sample access feature parameters corresponding to multiple historical access behaviors and their sample crawler results;

[0011] In the case that the current access behavior corresponding to the crawler identification result is a malicious crawler behavior, crawler resistance is performed on the current access behavior.

[0012] According to the technical solution provided by the present application, before inputting the access behavior parameter and the access feature parameter into the preset crawler recognition model, the method further includes:

[0013] Get sample access logs;

[0014] According to the sample access log, access behavior parameters corresponding to a plurality of sample access behaviors are obtained as sample access behavior parameters;

[0015] According to the sample access log, a sample access behavior feature matrix is ​​constructed, and according to the sample access behavior feature matrix, a sample access feature parameter corresponding to each of the sample access behaviors is obtained;

[0016] The sample access behavior parameters and sample access feature parameters corresponding to each of the sample access behaviors are used as model inputs, and the sample crawler results corresponding to each of the sample access behaviors are used as model outputs to perform model training to obtain the preset crawler recognition model.

[0017] According to the technical solution provided by the present application, the access data of the current access behavior includes a user agent; after obtaining the access data of the current access behavior in real time and before obtaining the access log, the method further includes:

[0018] If it is identified according to the user agent that the current access behavior includes a crawler tool, the current access behavior is directly determined to be a malicious crawler behavior.

[0019] According to the technical solution provided by the present application, the access data includes the access identifier, access type, access time, and request URL of the current access behavior; the access behavior parameters corresponding to the current access behavior are obtained according to the access data and the access log, including:

[0020] Extracting access data of historical access behaviors including the access identifier from the access log as reference access data;

[0021] According to the access type and access time, obtain historical access behaviors with the same access type and adjacent access time as the current access behavior from the reference access data, calculate the time interval between the access time of the historical access behavior and the access time of the current access behavior, and obtain the access interval corresponding to the current access behavior;

[0022] According to the request URL and the access time, an access request sequence consisting of the historical access behavior and the current access behavior corresponding to the access identifier is obtained from the reference access data and the access data of the current access behavior, and a behavior pattern corresponding to the current access behavior is obtained;

[0023] According to the access type and access time, the number of access behaviors of the same access type within a unit time is obtained from the reference access data and the access data of the current access behavior, and the access frequency corresponding to the current access behavior is obtained.

[0024] According to the technical solution provided by the present application, the step of constructing an access behavior feature matrix based on the updated access log includes:

[0025] Extract the access identifiers corresponding to all access behaviors in the updated access log, and obtain the access behavior parameters corresponding to all access behaviors in the updated access log;

[0026] The access behavior feature matrix is ​​constructed by taking the access identifiers corresponding to all access behaviors in the updated access log as matrix rows and taking the access behavior parameters corresponding to all access behaviors in the updated access log as matrix columns.

[0027] According to the technical solution provided by the present application, before constructing the access behavior feature matrix according to the updated access log, the method further includes:

[0028] Invalid data and noise data in the updated access log are screened, and the invalid data and noise data are removed from the updated access log.

[0029] According to the technical solution provided by the present application, the crawler resistance to the current access behavior includes:

[0030] Requesting to limit or block the current access behavior;

[0031] Sending an identity authentication instruction to the terminal that initiated the current access behavior;

[0032] The identity authentication information returned by the terminal is received, and when it is determined according to the identity authentication information that the current access behavior is a normal access behavior, the speed limiting or blocking request for the current access behavior is stopped.

[0033] In a second aspect, the present application provides an anti-crawler monitoring and processing device, comprising:

[0034] A first acquisition module is used to acquire access data of current access behavior in real time;

[0035] A second acquisition module is used to acquire access logs, and acquire access behavior parameters corresponding to the current access behavior according to the access data and the access logs; the access behavior parameters include access type, access interval, access frequency, and behavior pattern;

[0036] A construction module, used to update the access log based on the access data, and construct an access behavior feature matrix according to the updated access log, and obtain access feature parameters corresponding to the current access behavior from the access behavior feature matrix; the access feature parameters include crawler risk type, access frequency standard deviation, access frequency mean, and access time distribution sequence;

[0037] A prediction module is used to input the access behavior parameters and the access feature parameters into a preset crawler recognition model to obtain a crawler recognition result corresponding to the current access behavior output by the preset crawler recognition model; the preset crawler recognition model is obtained by model training based on sample access behavior parameters and sample access feature parameters corresponding to multiple historical access behaviors and their sample crawler results;

[0038] A processing module is used to perform crawler countermeasure on the current access behavior when the current access behavior corresponding to the crawler identification result is a malicious crawler behavior.

[0039] In a third aspect, the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method described above when executing the computer program.

[0040] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, and the computer program implements the method described above when executed by a processor.

[0041] Compared with the prior art, the beneficial effects of the present application are as follows: the anti-crawler monitoring and processing method, device, equipment and medium provided by the present application can quickly detect malicious crawler behavior and take corresponding measures by real-time monitoring of access behavior and predicting and identifying using machine learning models. This real-time response mechanism can effectively prevent crawlers from abusing server resources. In addition, after detecting malicious crawler behavior, various countermeasures such as speed limit, blocking request, verification code verification, etc. can be taken to protect server resources from multiple levels. This multi-level protection mechanism increases the difficulty of crawlers bypassing protection. Among them, using machine learning models to analyze and predict access behavior can effectively improve the accuracy and robustness of crawler detection. By continuously optimizing and training models, new crawler behavior patterns can be adapted to improve detection accuracy. In addition, the access log data is matrixed according to specific dimensions, and the crawler behavior characteristics are identified through matrix data analysis, which can supplement and verify the detection results of the machine learning model and provide double protection. Through the analysis of access behavior, normal users and crawlers can be distinguished. While ensuring security, requests from legitimate users can be responded to normally, avoiding interference to legitimate users and providing a good user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solution in this embodiment, the drawings required for use in the description of the embodiment will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0043] Figure 1 A flowchart of an anti-crawler monitoring and processing method provided in an embodiment of the present application;

[0044] Figure 2 A schematic diagram of an anti-crawler monitoring and processing device provided in an embodiment of the present application;

[0045] Figure 3 A schematic diagram of a computer system of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0046] In order to enable those skilled in the art to better understand the technical solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings. The description in this section is only exemplary and explanatory and should not have any limiting effect on the protection scope of the present application. Specifically, the described embodiments are only embodiments of a part of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work should fall within the scope of protection of the present application.

[0047] It should be noted that similar reference numerals and letters represent similar items in the following figures, so once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0048] In order to make the technical solution of the present application clearer and easier to understand, the anti-crawler monitoring and processing method provided in the embodiment of the present application is introduced below.

[0049] like Figure 1 As shown in the figure, this figure is a flow chart of an anti-crawler monitoring processing method provided by this embodiment, and the method includes the following steps:

[0050] S101, obtaining access data of current access behavior in real time;

[0051] Specifically, the access data of the current access behavior includes information such as access type, access time, access identifier, user agent, request URL, etc., wherein the access identifier may be an IP address. It is understandable that the access data may also include other parameters, which may be set and adjusted according to actual conditions, and are not specifically limited here.

[0052] For example, the current access behavior is as follows:

[0053] 127.0.0.1 - - [11 / Jul / 2024:12:00:01 +0000] "GET / index.html HTTP / 1.1" 200 1234 "-" "Mozilla / 5.0 (Windows NT 10.0; Win64; x64) AppleWebKit / 537.36(KHTML, like Gecko) Chrome / 88.0.4324.150 Safari / 537.36"; The access time extracted from it is: 10 / Jul / 2024:11:05:42 +0000;

[0054] Extract the access data of the current access behavior from it:

[0055] The access identifier (IP address) is: 127.0.0.1;

[0056] Access type: GET (request data)

[0057] User agent: Mozilla / 5.0 (Windows NT 10.0; Win64; x64) AppleWebKit / 537.36 (KHTML, like Gecko) Chrome / 88.0.4324.150 Safari / 537.36

[0058] The request URL is: / index.html.

[0059] S102, obtaining an access log, and obtaining access behavior parameters corresponding to the current access behavior according to the access data and the access log; the access behavior parameters include access type, access interval, access frequency, and behavior pattern;

[0060] Specifically, to obtain access logs, you can use log management tools (such as ELK Stack) or write custom scripts to collect and process log data:

[0061] import pandas as pd

[0062] import numpy as np

[0063] from datetime import datetime

[0064] from sklearn.ensemble import RandomForestClassifier

[0065] import joblib

[0066] # Load the trained model

[0067] model = joblib.load('trained_model.pkl')

[0068] # Real-time monitoring function

[0069] def monitor_logs(log_entry):

[0070] # Assume log_entry is a dictionary containing new access data

[0071] log_entry["timestamp"] = pd.to_datetime(log_entry["timestamp"],format='%d / %b / %Y:%H:%M:%S %z')

[0072] return log_entry

[0073] # Example log entry

[0074] new_log_entry = {

[0075] "ip": "127.0.0.1",

[0076] "timestamp": "11 / Jul / 2024:12:00:01 +0000",

[0077] "url": " / index.html",

[0078] "user_agent": "Mozilla / 5.0 (Windows NT 10.0; Win64; x64)AppleWebKit / 537.36 (KHTML, like Gecko) Chrome / 88.0.4324.150 Safari / 537.36"

[0079] }

[0080] new_log_entry = monitor_logs(new_log_entry).

[0081] Then, the corresponding access type can be extracted from the access data of the current access behavior; further, the access data of the historical access behavior containing the access identifier is extracted from the access log as reference access data, and according to the access type and access time, the historical access behavior with the same access type and adjacent access time as the current access behavior is obtained from the reference access data, and the time interval between the access time of the historical access behavior and the access time of the current access behavior is calculated to obtain the access interval corresponding to the current access behavior; according to the request URL and access time, the access request sequence consisting of the historical access behavior corresponding to the access identifier and the current access behavior is obtained from the reference access data and the access data of the current access behavior to obtain the behavior pattern corresponding to the current access behavior; according to the access type and access time, the number of access behaviors of the same access type per unit time is obtained from the reference access data and the access data of the current access behavior to obtain the access frequency corresponding to the current access behavior, so as to obtain the access behavior parameter corresponding to the current access behavior.

[0082] S103, updating the access log based on the access data, and constructing an access behavior feature matrix according to the updated access log, and obtaining access feature parameters corresponding to the current access behavior from the access behavior feature matrix; the access feature parameters include crawler risk type, access frequency standard deviation, access frequency mean, and access time distribution sequence;

[0083] Specifically, the access data of the current access behavior is merged with the access log to update the access log. The updated access log includes not only the access data of the previous historical access behavior but also the access data of the current access behavior. At this time, the access identifiers corresponding to all access behaviors in the updated access log are extracted, and the access behavior parameters corresponding to all access behaviors in the updated access log are obtained. The access identifiers corresponding to all access behaviors in the updated access log are used as matrix rows, and the access behavior parameters corresponding to all access behaviors in the updated access log are used as matrix columns to construct the access behavior feature matrix, and the access feature parameters corresponding to the current access behavior are obtained from the access behavior feature matrix. Among them, the access feature parameters include crawler risk type, access frequency standard deviation, access frequency mean, and access time distribution sequence.

[0084] The access frequency standard deviation can be obtained by calculating the standard deviation of the access frequency of an IP address in each time period. The access frequency standard deviation is used to characterize the variability of the access pattern. The higher the standard deviation, the more abnormal the access behavior is, which may be a crawler; for example:

[0085] # Calculate the standard deviation of the number of visits for each IP address

[0086] std_dev = df.std(axis=1)

[0087] print(std_dev)

[0088] If the access frequencies of different IP addresses in different time periods are shown in Table 1:

[0089] Table 1

[0090]

[0091] By calculating the standard deviation of the access frequency of each IP address, we can find that:

[0092] The standard deviation of 127.0.0.1 is 0.577 (high, probably a crawler behavior)

[0093] The standard deviation for 192.168.1.1 is 0 (low, probably normal user behavior).

[0094] The access frequency mean can be obtained by calculating the average value of the access frequency in a certain period of time, wherein the specific calculation method of the access frequency is the same as the calculation method of the access frequency in the above-mentioned calculation of the access behavior parameter, and will not be repeated here.

[0095] The access time distribution sequence is obtained by obtaining the time of all access behaviors of a certain IP address within a certain period of time, and arranging the obtained sequence in chronological order, so as to characterize the time distribution of the occurrence of the access behaviors.

[0096] The crawler risk type is determined based on the access frequency standard deviation, access frequency mean, and access time distribution sequence, as shown in Table 2, and is used to characterize the preliminary judgment of the crawler results.

[0097] Table 2

[0098]

[0099] S104, input the access behavior parameter and the access feature parameter into a preset crawler recognition model, and obtain a crawler recognition result corresponding to the current access behavior output by the preset crawler recognition model; the preset crawler recognition model is obtained by model training based on sample access behavior parameters and sample access feature parameters corresponding to multiple historical access behaviors and their sample crawler results;

[0100] Specifically, a sample access log is obtained in advance, and access behavior parameters corresponding to multiple sample access behaviors are obtained according to the sample access log as sample access behavior parameters. A sample access behavior feature matrix is ​​constructed according to the sample access log, and sample access feature parameters corresponding to each sample access behavior are obtained according to the sample access behavior feature matrix. The sample access behavior parameters and sample access feature parameters corresponding to each sample access behavior are used as model inputs, and the sample crawler results corresponding to each sample access behavior are used as model outputs to perform model training to obtain the preset crawler recognition model. Then, the access behavior parameters and the access feature parameters are input into the preset crawler recognition model to obtain the crawler recognition result corresponding to the current access behavior output by the preset crawler recognition model. The crawler recognition result includes "is malicious crawler behavior" and "is not malicious crawler behavior", which can be represented by "1" and "0" respectively.

[0101] S105: When the current access behavior corresponding to the crawler identification result is a malicious crawler behavior, perform crawler countermeasure on the current access behavior.

[0102] Specifically, when the current access behavior corresponding to the crawler identification result is a malicious crawler behavior, a speed limit or blocking request is made to the current access behavior; at the same time, an identity authentication instruction is sent to the terminal that initiates the current access behavior, and the identity authentication information returned by the terminal is received. When it is determined that the current access behavior is a normal access behavior based on the identity authentication information, the speed limit or blocking request for the current access behavior is stopped to ensure the smooth progress of the normal access behavior.

[0103] Further on the basis of the above embodiment, before inputting the access behavior parameter and the access feature parameter into the preset crawler recognition model, the method further includes:

[0104] Get sample access logs;

[0105] According to the sample access log, access behavior parameters corresponding to a plurality of sample access behaviors are obtained as sample access behavior parameters;

[0106] According to the sample access log, a sample access behavior feature matrix is ​​constructed, and according to the sample access behavior feature matrix, a sample access feature parameter corresponding to each of the sample access behaviors is obtained;

[0107] The sample access behavior parameters and sample access feature parameters corresponding to each of the sample access behaviors are used as model inputs, and the sample crawler results corresponding to each of the sample access behaviors are used as model outputs to perform model training to obtain the preset crawler recognition model.

[0108] Specifically, access logs within a certain historical time period are obtained as sample access logs. The historical time period can be selected according to specific circumstances, and adjustments can be made when necessary to improve the prediction accuracy of the model, which is not specifically limited here. According to the sample access logs, access behavior parameters corresponding to multiple sample access behaviors are obtained as sample access behavior parameters; according to the sample access logs, a sample access behavior feature matrix is ​​constructed, and sample access feature parameters corresponding to each sample access behavior are obtained according to the sample access behavior feature matrix; a certain proportion of sample access behavior parameters, sample access feature parameters and corresponding sample crawler results are selected as training sets, and the rest are used as test sets; the training set is input into the original prediction model for model training, and the training results are cross-validated based on the test set and the matrix dimensions are adjusted or new features are added or model parameters are optimized until the prediction accuracy is greater than the preset threshold to ensure the accuracy and robustness of the model. Among them, commonly used original prediction models include random forests, support vector machines (SVMs), neural networks, etc., which are not specifically limited here.

[0109] Further on the basis of the above embodiment, the access data of the current access behavior includes a user agent; after obtaining the access data of the current access behavior in real time and before obtaining the access log, the method further includes:

[0110] If it is identified according to the user agent that the current access behavior includes a crawler tool, the current access behavior is directly determined to be a malicious crawler behavior.

[0111] Specifically, by analyzing the user agent, known crawler tools can be identified. If the current access behavior is identified to contain a crawler tool according to the user agent, the current access behavior is directly determined to be a malicious crawler behavior, and crawler resistance is directly performed on the current access behavior. For example, if the user agent contains "Googlebot", it can be determined that the request comes from the Google crawler.

[0112] Further on the basis of the above embodiment, the access data includes the access identifier, access type, access time, and request URL of the current access behavior; the access behavior parameters corresponding to the current access behavior are obtained according to the access data and the access log, including:

[0113] Extracting access data of historical access behaviors including the access identifier from the access log as reference access data;

[0114] Specifically, using the access identifier (IP address) as an index, all access behaviors in the access log are grouped by IP address, and access data of historical access behaviors containing the access identifier, that is, historical access behavior data of the IP address that initiated the current access behavior, is extracted from the access log as reference access data.

[0115] According to the access type and access time, obtain historical access behaviors with the same access type and adjacent access time as the current access behavior from the reference access data, calculate the time interval between the access time of the historical access behavior and the access time of the current access behavior, and obtain the access interval corresponding to the current access behavior;

[0116] Specifically, the access interval refers to the time difference between consecutive access requests from the same IP address. The timestamp in the log is converted into a datetime object, and historical access behaviors with the same access type and adjacent access time as the current access behavior are obtained from the reference access data. The access interval can be obtained by calculating the time difference between the two.

[0117] For example, given the following log entry:

[0118] 127.0.0.1 - - [10 / Jul / 2024:11:05:42 +0000] "GET / index.html HTTP / 1.1"200 1234 "-" "Mozilla / 5.0 (Windows NT 10.0; Win64; x64) AppleWebKit / 537.36(KHTML, like Gecko) Chrome / 88.0.4324.150 Safari / 537.36"

[0119] 127.0.0.1 - - [10 / Jul / 2024:11:05:43 +0000] "GET / index.html HTTP / 1.1"200 1234 "-" "Mozilla / 5.0 (Windows NT 10.0; Win64; x64) AppleWebKit / 537.36(KHTML, like Gecko) Chrome / 88.0.4324.150 Safari / 537.36"

[0120] 192.168.1.1 - - [10 / Jul / 2024:11:06:00 +0000] "GET / about.html HTTP / 1.1" 200 5678 "-" "Mozilla / 5.0 (Windows NT 10.0; Win64; x64) AppleWebKit / 537.36 (KHTML, like Gecko) Chrome / 88.0.4324.150 Safari / 537.36"

[0121] By calculating the time difference, we can find that the interval between the two accesses to 127.0.0.1 is 1 second, and there is no interval because 192.168.1.1 has no other requests.

[0122] In addition to the GET request type in the above example, there are other request types, such as:

[0123] POST request: Usually used to submit data to the server, such as form submission or file upload. Unlike GET request, the data of POST request is contained in the request body instead of being attached to the URL.

[0124] PUT request: mainly used to update resources on the server. The client will send the entire resource content to the server, and the server will update the corresponding resources according to the request content.

[0125] DELETE request: used to request the server to delete the specified resources.

[0126] HEAD request: Similar to the GET request, but the server does not return the message body when responding, only the response header, which is often used to check the meta information or status of the resource.

[0127] OPTIONS request: used to describe the communication options of the target resource. The client can send an OPTIONS request to the resource specified by the server to understand the server's support for the resource, such as supported HTTP methods and header information.

[0128] PATCH request: used to partially modify resources. Unlike the PUT request, the PATCH request allows the client to send only the part that needs to be modified, rather than the entire resource.

[0129] TRACE request: mainly used to diagnose and trace the request chain along the path to the target resource. The server will return the received request to the client intact, mainly used for testing and diagnosis.

[0130] According to the request URL and the access time, an access request sequence consisting of the historical access behavior and the current access behavior corresponding to the access identifier is obtained from the reference access data and the access data of the current access behavior, and a behavior pattern corresponding to the current access behavior is obtained;

[0131] Specifically, the behavior pattern is obtained by analyzing the user's access path and sequence by recording the request path sequence of the IP address, and according to the request URL and access time, the access request sequence consisting of the historical access behavior and the current access behavior corresponding to the access identifier is obtained from the reference access data and the access data of the current access behavior, and the behavior pattern corresponding to the current access behavior is obtained, so that it can be determined whether the user's access pattern conforms to the normal user's behavior. For example, a user may access pages in a specific order, while a crawler may randomly or systematically access all pages.

[0132] According to the access type and access time, the number of access behaviors of the same access type within a unit time is obtained from the reference access data and the access data of the current access behavior, and the access frequency corresponding to the current access behavior is obtained.

[0133] Specifically, the access frequency indicates the number of visits to a certain IP address within a unit time. According to the access type and access time, the number of access behaviors of the same access type within a unit time is obtained from the reference access data and the access data of the current access behavior, and the access frequency corresponding to the current access behavior is obtained. If the access frequency of a certain IP address is abnormally high, it may be a crawler behavior.

[0134] Further on the basis of the above embodiment, the step of constructing an access behavior feature matrix according to the updated access log includes:

[0135] Extract the access identifiers corresponding to all access behaviors in the updated access log, and obtain the access behavior parameters corresponding to all access behaviors in the updated access log;

[0136] The access behavior feature matrix is ​​constructed by taking the access identifiers corresponding to all access behaviors in the updated access log as matrix rows and taking the access behavior parameters corresponding to all access behaviors in the updated access log as matrix columns.

[0137] Specifically, the access data of the current access behavior is merged with the access log to update the access log. The updated access log includes not only the access data of previous historical access behaviors but also the access data of current access behaviors. The access identifiers corresponding to all access behaviors in the updated access log are used as matrix rows, and the access behavior parameters corresponding to all access behaviors in the updated access log are used as matrix columns to construct the access behavior feature matrix.

[0138] For example, the updated access log includes:

[0139] 127.0.0.1 - - [10 / Jul / 2024:11:05:42 +0000] "GET / index.html HTTP / 1.1"200 1234 "-" "Mozilla / 5.0 (Windows NT 10.0; Win64; x64) AppleWebKit / 537.36(KHTML, like Gecko) Chrome / 88.0.4324.150 Safari / 537.36"

[0140] 127.0.0.1 - - [10 / Jul / 2024:11:10:43 +0000] "GET / index.html HTTP / 1.1"200 1234 "-" "Mozilla / 5.0 (Windows NT 10.0; Win64; x64) AppleWebKit / 537.36(KHTML, like Gecko) Chrome / 88.0.4324.150 Safari / 537.36"

[0141] 192.168.1.1 - - [10 / Jul / 2024:11:06:00 +0000] "GET / about.html HTTP / 1.1" 200 5678 "-" "Mozilla / 5.0 (Windows NT 10.0; Win64; x64) AppleWebKit / 537.36 (KHTML, like Gecko) Chrome / 88.0.4324.150 Safari / 537.36"

[0142] Take the access frequency in the access behavior parameter as an example, where the matrix rows represent different IP addresses, the matrix columns represent time periods (such as every minute), and the matrix values ​​represent the access frequency of a certain IP address in a specific time period. The specific implementation is as follows:

[0143] import pandas as pd

[0144] from collections import defaultdict

[0145] # Sample access log data

[0146] log_entries = [

[0147] {"ip": "127.0.0.1", "timestamp": "10 / Jul / 2024:11:05:42 +0000", "url": " / index.html"},

[0148] {"ip": "127.0.0.1", "timestamp": "10 / Jul / 2024:11:10:43 +0000", "url": " / index.html"},

[0149] {"ip": "192.168.1.1", "timestamp": "10 / Jul / 2024:11:06:00 +0000", "url": " / about.html"} ]

[0151] # Convert timestamp to datetime object and round to minutes

[0152] for entry in log_entries:

[0153] entry["timestamp"] = pd.to_datetime(entry["timestamp"], format='%d / %b / %Y:%H:%M:%S %z').round('min')

[0154] # Build the access matrix

[0155] matrix = defaultdict(lambda: defaultdict(int))

[0156] for entry in log_entries:

[0157] ip = entry["ip"]

[0158] time = entry["timestamp"]

[0159] matrix[ip][time]+= 1

[0160] # Convert dictionary to DataFrame

[0161] df = pd.DataFrame(matrix).fillna(0).astype(int)

[0162] print(df).

[0163] Further on the basis of the above embodiment, before constructing the access behavior feature matrix according to the updated access log, the method further includes:

[0164] Invalid data and noise data in the updated access log are screened, and the invalid data and noise data are removed from the updated access log.

[0165] Specifically, invalid data and noise data include: repeated requests: repeated requests initiated by the same IP address within a short period of time, false positive data: invalid log entries caused by incorrect configuration or anomalies, robot traffic: common robot requests identified by simple rules.

[0166] Screening the invalid data and noise data in the updated access log and removing the invalid data and noise data from the updated access log can be called data cleaning. The specific steps of data cleaning are: removing duplicate requests: using a unique identifier (IP address + request URL + timestamp) to deduplicate, filtering false positive data: removing invalid entries based on the validity rules of the log format and content, identifying and marking robot traffic: using known robot user agent identifiers to filter or mark these traffics.

[0167] For example, there are the following access logs:

[0168] 127.0.0.1 - - [10 / Jul / 2024:11:05:42 +0000] "GET / index.html HTTP / 1.1"200 1234 "-" "Mozilla / 5.0 (Windows NT 10.0; Win64; x64) AppleWebKit / 537.36(KHTML, like Gecko) Chrome / 88.0.4324.150 Safari / 537.36"

[0169] 127.0.0.1 - - [10 / Jul / 2024:11:05:43 +0000] "GET / index.html HTTP / 1.1"200 1234 "-" "Mozilla / 5.0 (Windows NT 10.0; Win64; x64) AppleWebKit / 537.36(KHTML, like Gecko) Chrome / 88.0.4324.150 Safari / 537.36"

[0170] 127.0.0.2 - - [10 / Jul / 2024:11:05:45 +0000] "GET / index.html HTTP / 1.1"200 1234 "-" "Googlebot / 2.1 (+http: / / www.google.com / bot.html)"

[0171] 192.168.1.1 - - [10 / Jul / 2024:11:06:00 +0000] "GET / about.html HTTP / 1.1" 200 5678 "-" "Mozilla / 5.0 (Windows NT 10.0; Win64; x64) AppleWebKit / 537.36 (KHTML, like Gecko) Chrome / 88.0.4324.150 Safari / 537.36"

[0172] The access log after data cleaning is as follows:

[0173] 127.0.0.1 - - [10 / Jul / 2024:11:05:42 +0000] "GET / index.html HTTP / 1.1"200 1234 "-" "Mozilla / 5.0 (Windows NT 10.0; Win64; x64) AppleWebKit / 537.36(KHTML, like Gecko) Chrome / 88.0.4324.150 Safari / 537.36"

[0174] 192.168.1.1 - - [10 / Jul / 2024:11:06:00 +0000] "GET / about.html HTTP / 1.1" 200 5678 "-" "Mozilla / 5.0 (Windows NT 10.0; Win64; x64) AppleWebKit / 537.36 (KHTML, like Gecko) Chrome / 88.0.4324.150 Safari / 537.36"

[0175] After deduplication, false positive filtering, and robot traffic identification, the final valid log data is obtained.

[0176] Further on the basis of the above embodiment, the crawler resistance to the current access behavior includes:

[0177] Requesting to limit or block the current access behavior;

[0178] Sending an identity authentication instruction to the terminal that initiated the current access behavior;

[0179] The identity authentication information returned by the terminal is received, and when it is determined according to the identity authentication information that the current access behavior is a normal access behavior, the speed limiting or blocking request for the current access behavior is stopped.

[0180] Specifically, in order to protect requests involving sensitive information or encrypted data, it is necessary to add multiple levels of security protection measures, such as encrypted transmission, permission verification, etc. When malicious crawler behavior is detected, timely measures are taken to protect the security of sensitive information and prevent data leakage. The specific implementation steps are as follows:

[0181] Encrypted transmission: Ensure that all communications involving sensitive information use encrypted transmission protocols (such as HTTPS). Use SSL / TLS certificates to encrypt data transmission to prevent man-in-the-middle attacks and data theft.

[0182] # Assume that we use the Flask framework to implement a simple HTTPS server

[0183] from flask import Flask, request, jsonify

[0184] from OpenSSL import SSL

[0185] app = Flask(__name__)

[0186] # Using SSL / TLS Certificates

[0187] context = SSL.Context(SSL.SSLv23_METHOD)

[0188] context.use_privatekey_file('path / to / privatekey.pem')

[0189] context.use_certificate_file('path / to / certificate.pem')

[0190] @app.route(' / sensitive-data', methods=['GET'])

[0191] def sensitive_data():

[0192] return jsonify({"data": "This is sensitive data."})

[0193] if __name__ == '__main__':

[0194] app.run(ssl_context=context)

[0195] Permission verification: When processing requests involving sensitive information, strict permission verification is performed. Only authenticated and authorized users can access this data. JWT (JSON Web Token) can be used to implement user authentication and authorization.

[0196] import jwt

[0197] from functools import wraps

[0198] from flask import request, jsonify

[0199] SECRET_KEY = 'your_secret_key'

[0200] # JWT Verification Decorator

[0201] def token_required(f):

[0202] @wraps(f)

[0203] def decorated( args, kwargs):

[0204] token = request.headers.get('Authorization')

[0205] if not token:

[0206] return jsonify({"message": "Token is missing!"}), 403

[0207] try:

[0208] jwt.decode(token, SECRET_KEY, algorithms=['HS256'])

[0209] except jwt.ExpiredSignatureError:

[0210] return jsonify({"message": "Token has expired!"}), 403

[0211] except jwt.InvalidTokenError:

[0212] return jsonify({"message": "Invalid token!"}), 403

[0213] return f( args, kwargs

[0214] return decorated

[0215] @app.route(' / sensitive-data', methods=['GET'])

[0216] @token_required

[0217] def sensitive_data():

[0218] return jsonify({"data": "This is sensitive data."})

[0219] Detect malicious crawler behavior and take measures: When malicious crawler behavior is detected, the protection mechanism is triggered to take timely measures to protect the security of sensitive information.

[0220] # Example log entry

[0221] log_entries = [

[0222] {"ip": "127.0.0.1", "timestamp": "11 / Jul / 2024:12:00:01 +0000", "url": " / sensitive-data"},

[0223] {"ip": "127.0.0.1", "timestamp": "11 / Jul / 2024:12:00:02 +0000", "url": " / sensitive-data"},

[0224] {"ip": "192.168.1.1", "timestamp": "11 / Jul / 2024:12:00:03 +0000", "url": " / sensitive-data"}

[0226] # Real-time monitoring function

[0227] def monitor_logs(log_entry):

[0228] log_entry["timestamp"] = pd.to_datetime(log_entry["timestamp"],format='%d / %b / %Y:%H:%M:%S %z')

[0229] return log_entry

[0230] # Extract features

[0231] def extract_features(log_entry):

[0232] ip = log_entry["ip"]

[0233] current_time = log_entry["timestamp"]

[0234] if ip in last_visit:

[0235] interval = (current_time - last_visit[ip]).total_seconds()

[0236] else:

[0237] interval = 0

[0238] last_visit[ip] = current_time

[0239] ​request_count = frequency[ip] if ip in frequency else 1

[0240] unique_paths = len(interaction[ip]) if ip in interaction else 1

[0241] features = [interval, request_count, unique_paths]

[0242] return np.array(features).reshape(1, -1)

[0243] # Detecting Malicious Crawler Behavior

[0244] def detect_and_protect(log_entry):

[0245] new_log_entry = monitor_logs(log_entry)

[0246] new_features = extract_features(new_log_entry)

[0247] is_bot = model.predict(new_features)[0]

[0248] if is_bot:

[0249] print("Detected bot activity. Blocking access to sensitivedata.")

[0250] # Take countermeasures, such as blocking requests or requiring users to complete a captcha

[0251] else:

[0252] print("Legitimate user. Allowing access to sensitive data.")

[0253] # Example new access log entry

[0254] new_log_entry = {"ip": "127.0.0.1", "timestamp": "11 / Jul / 2024:12:00:04 +0000", "url": " / sensitive-data"}

[0255] detect_and_protect(new_log_entry)

[0256] Suppose we monitor the following new access data in real time:

[0257] 127.0.0.1 - - [11 / Jul / 2024:12:00:01 +0000] "GET / sensitive-data HTTP / 1.1" 200 1234 "-" "Mozilla / 5.0"

[0258] 127.0.0.1 - - [11 / Jul / 2024:12:00:02 +0000] "GET / sensitive-data HTTP / 1.1" 200 1234 "-" "Mozilla / 5.0"

[0259] 192.168.1.1 - - [11 / Jul / 2024:12:00:03 +0000] "GET / sensitive-dataHTTP / 1.1" 200 5678 "-" "Mozilla / 5.0"

[0260] Through feature extraction and analysis, we calculated that the model output of 127.0.0.1 is 1, indicating that crawler behavior is detected. Therefore, we take measures to block access requests from this IP address to protect the security of sensitive information. For the legitimate user 192.168.1.1, because the model predicts an output of 0, indicating that its behavior is normal, we allow it to access sensitive data normally.

[0261] Through multi-layered security protection measures (such as encrypted transmission and permission verification) as well as real-time crawler detection and response mechanisms, we are able to effectively protect sensitive information on the server and prevent data leakage.

[0262] The embodiment of the present application provides an anti-crawler monitoring and processing method, which obtains the access data of the current access behavior in real time; obtains the access log, obtains the access behavior parameters corresponding to the current access behavior according to the access data and the access log; updates the access log based on the access data, and constructs the access behavior feature matrix according to the updated access log, and obtains the access feature parameters corresponding to the current access behavior from the access behavior feature matrix; inputs the access behavior parameters and the access feature parameters to the preset crawler recognition model obtained by model training based on the sample access behavior parameters and the sample access feature parameters corresponding to multiple historical access behaviors and their sample crawler results, and obtains the crawler recognition result corresponding to the current access behavior output by the preset crawler recognition model; when the current access behavior corresponding to the crawler recognition result is a malicious crawler behavior, the current access behavior is subjected to crawler confrontation. By monitoring the access behavior in real time and using the machine learning model for prediction and identification, malicious crawler behavior can be quickly detected and corresponding measures can be taken. This real-time response mechanism can effectively prevent the abuse of server resources by crawlers. In addition, after detecting malicious crawler behavior, various countermeasures such as speed limit, blocking request, verification code verification, etc. can be taken to protect server resources from multiple levels. This multi-level protection mechanism increases the difficulty of crawlers bypassing protection. Among them, using machine learning models to analyze and predict access behaviors can effectively improve the accuracy and robustness of crawler detection. By continuously optimizing and training the model, it can adapt to new crawler behavior patterns and improve detection accuracy. In addition, the access log data is organized into a matrix according to specific dimensions, and the crawler behavior characteristics are identified through matrix data analysis, which can supplement and verify the detection results of the machine learning model and provide double protection. Through the analysis of access behavior, normal users and crawlers can be distinguished. While ensuring security, requests from legitimate users can be responded to normally, avoiding interference with legitimate users and providing a good user experience.

[0263] Combination of the above Figure 1 The anti-crawler monitoring and processing method provided in the embodiment of the present application is introduced in detail. The anti-crawler monitoring and processing device, electronic device and computer-readable storage medium provided in the embodiment of the present application will be introduced in conjunction with the accompanying drawings.

[0264] like Figure 2 As shown in the figure, this figure is a schematic diagram of the anti-crawler monitoring and processing device provided by the present application, and the device includes:

[0265] The first acquisition module 201 is used to acquire access data of current access behavior in real time;

[0266] The second acquisition module 202 is used to acquire the access log, and acquire the access behavior parameters corresponding to the current access behavior according to the access data and the access log; the access behavior parameters include access type, access interval, access frequency, and behavior mode;

[0267] A construction module 203 is used to update the access log based on the access data, and to construct an access behavior feature matrix according to the updated access log, and to obtain access feature parameters corresponding to the current access behavior from the access behavior feature matrix; the access feature parameters include crawler risk type, access frequency standard deviation, access frequency mean, and access time distribution sequence;

[0268] Prediction module 204, used for inputting the access behavior parameter and the access feature parameter into a preset crawler recognition model, and obtaining a crawler recognition result corresponding to the current access behavior output by the preset crawler recognition model; the preset crawler recognition model is obtained by model training based on sample access behavior parameters and sample access feature parameters corresponding to multiple historical access behaviors and their sample crawler results;

[0269] The processing module 205 is used to perform crawler resistance on the current access behavior when the current access behavior corresponding to the crawler identification result is a malicious crawler behavior.

[0270] The anti-crawler monitoring processing device provided in the embodiment of the present application may correspond to executing the anti-crawler monitoring processing method described in the embodiment of the present application, and the above functions of each module of the device correspond to implementing Figure 1 For the sake of brevity, the corresponding process of the method shown will not be repeated here.

[0271] An embodiment of the present application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the anti-crawler monitoring and processing method as described in the above embodiment is implemented.

[0272] like Figure 3 As shown, the computer system 300 of the electronic device includes a CPU 301, which can perform various appropriate actions and processes according to the program stored in the ROM 302 or the program loaded from the storage part 308 to the RAM 303. In the RAM 303, various programs and data required for system operation are also stored. The CPU 301, the ROM 302 and the RAM 303 are connected to each other through a bus 304. The I / O interface 305 is also connected to the bus 304. Among them, CPU 301 represents a central processing unit, ROM 302 represents a read-only memory, RAM 303 represents a random access memory, and I / O represents input / output.

[0273] The following components are connected to the I / O interface 305: an input section 306 including a keyboard, a mouse, etc.; an output section 307 including a cathode ray tube, a liquid crystal display, etc., and a speaker, etc.; a storage section 308 including a hard disk, etc.; and a communication section 309 including a network interface card such as a LAN card, a modem, etc. The communication section 309 performs communication processing via a network such as the Internet. A drive 310 is also connected to the I / O interface 305 as needed. A removable medium 311, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 310 as needed, so that a computer program read therefrom is installed into the storage section 308 as needed.

[0274] In particular, the process of the anti-crawler monitoring processing method described in the above embodiment can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable storage medium, and the computer program contains program code for executing the anti-crawler monitoring processing method described in the above embodiment. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 309, and / or installed from the removable medium 311. When the computer program is executed by CPU301, the above functions defined in the present computer system 300 are executed.

[0275] An embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the anti-crawler monitoring and processing method as described in the above embodiment is implemented.

[0276] Specifically, the computer-readable storage medium may be included in the electronic device described in the above embodiment; or it may exist independently without being assembled into the electronic device. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed by an electronic device, the electronic device implements the anti-crawler monitoring processing method described in the above embodiment.

[0277] It should be noted that the computer-readable storage medium shown in the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory, a read-only memory, an erasable programmable read-only memory, an optical fiber, a portable compact disk read-only memory, an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above.

[0278] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and its core ideas of this application. The above is only the preferred implementation method of this application. It should be pointed out that due to the limitations of textual expression and the objective existence of infinite specific structures, ordinary technicians in this technical field can make several improvements, modifications or changes without departing from the principles of the present invention, and can also combine the above technical features in an appropriate manner; these improvements, modifications, changes or combinations, or the direct application of the inventive concept and technical solution to other occasions without improvement, should be regarded as the scope of protection of this application.

Claims

1. An anti-crawler monitoring and processing method, characterized in that: include: Acquire access data of current access behavior in real time; the access data includes access identification, access type, access time, and request URL of the current access behavior; Obtaining the access log, and obtaining the access behavior parameters corresponding to the current access behavior according to the access data and the access log; the access behavior parameters include access type, access interval, access frequency, and behavior mode; obtaining the access behavior parameters corresponding to the current access behavior according to the access data and the access log includes: Extracting access data of historical access behaviors including the access identifier from the access log as reference access data; According to the access type and access time, obtain historical access behaviors with the same access type and adjacent access time as the current access behavior from the reference access data, calculate the time interval between the access time of the historical access behavior and the access time of the current access behavior, and obtain the access interval corresponding to the current access behavior; According to the request URL and the access time, an access request sequence consisting of the historical access behavior and the current access behavior corresponding to the access identifier is obtained from the reference access data and the access data of the current access behavior, and a behavior pattern corresponding to the current access behavior is obtained; According to the access type and access time, the number of access behaviors of the same access type in a unit time is obtained from the reference access data and the access data of the current access behavior, and the access frequency corresponding to the current access behavior is obtained; The access log is updated based on the access data, and an access behavior feature matrix is ​​constructed according to the updated access log, and access feature parameters corresponding to the current access behavior are obtained from the access behavior feature matrix; the access feature parameters include crawler risk type, access frequency standard deviation, access frequency mean, and access time distribution sequence; the crawler risk type is determined based on the access frequency standard deviation, access frequency mean, and access time distribution sequence, and is used to characterize the preliminary judgment of the crawler result, and the crawler risk type includes low-frequency normal requests, high-frequency suspected crawler requests, normal requests during concentrated periods, evenly distributed suspected crawler requests, and sudden peak requests; The step of constructing an access behavior feature matrix according to the updated access logs includes: Extract the access identifiers corresponding to all access behaviors in the updated access log, and obtain the access behavior parameters corresponding to all access behaviors in the updated access log; The access behavior feature matrix is ​​constructed by using access identifiers corresponding to all access behaviors in the updated access log as matrix rows and access behavior parameters corresponding to all access behaviors in the updated access log as matrix columns; Input the access behavior parameter and the access feature parameter into a preset crawler recognition model to obtain a crawler recognition result corresponding to the current access behavior output by the preset crawler recognition model; the preset crawler recognition model is obtained by model training based on sample access behavior parameters and sample access feature parameters corresponding to multiple historical access behaviors and their sample crawler results; In the case that the current access behavior corresponding to the crawler identification result is a malicious crawler behavior, crawler resistance is performed on the current access behavior.

2. The method according to claim 1, characterized in that Before inputting the access behavior parameter and the access feature parameter into the preset crawler recognition model, the method further includes: Get sample access logs; According to the sample access log, access behavior parameters corresponding to a plurality of sample access behaviors are obtained as sample access behavior parameters; According to the sample access log, a sample access behavior feature matrix is ​​constructed, and according to the sample access behavior feature matrix, a sample access feature parameter corresponding to each of the sample access behaviors is obtained; The sample access behavior parameters and sample access feature parameters corresponding to each of the sample access behaviors are used as model inputs, and the sample crawler results corresponding to each of the sample access behaviors are used as model outputs to perform model training to obtain the preset crawler recognition model.

3. The method according to claim 1, characterized in that The access data of the current access behavior includes a user agent; after obtaining the access data of the current access behavior in real time and before obtaining the access log, the method further includes: If it is identified according to the user agent that the current access behavior includes a crawler tool, the current access behavior is directly determined to be a malicious crawler behavior.

4. The method according to claim 3, characterized in that Before constructing the access behavior feature matrix according to the updated access log, the method further includes: Invalid data and noise data in the updated access log are screened, and the invalid data and noise data are removed from the updated access log.

5. The method according to any one of claims 1 to 4, characterized in that: The performing crawler resistance on the current access behavior includes: Requesting to limit or block the current access behavior; Sending an identity authentication instruction to the terminal that initiated the current access behavior; The identity authentication information returned by the terminal is received, and when it is determined according to the identity authentication information that the current access behavior is a normal access behavior, the speed limiting or blocking request for the current access behavior is stopped.

6. An anti-crawler monitoring and processing device, characterized in that: include: The first acquisition module is used to acquire access data of the current access behavior in real time; the access data includes the access identifier, access type, access time, and request URL of the current access behavior; The second acquisition module is used to acquire the access log, and acquire the access behavior parameters corresponding to the current access behavior according to the access data and the access log; the access behavior parameters include access type, access interval, access frequency, and behavior mode; acquiring the access behavior parameters corresponding to the current access behavior according to the access data and the access log includes: Extracting access data of historical access behaviors including the access identifier from the access log as reference access data; According to the access type and access time, obtain historical access behaviors with the same access type and adjacent access time as the current access behavior from the reference access data, calculate the time interval between the access time of the historical access behavior and the access time of the current access behavior, and obtain the access interval corresponding to the current access behavior; According to the request URL and the access time, an access request sequence consisting of the historical access behavior and the current access behavior corresponding to the access identifier is obtained from the reference access data and the access data of the current access behavior, and a behavior pattern corresponding to the current access behavior is obtained; According to the access type and access time, the number of access behaviors of the same access type in a unit time is obtained from the reference access data and the access data of the current access behavior, and the access frequency corresponding to the current access behavior is obtained; A construction module, used to update the access log based on the access data, and to construct an access behavior feature matrix according to the updated access log, and to obtain access feature parameters corresponding to the current access behavior from the access behavior feature matrix; the access feature parameters include crawler risk type, access frequency standard deviation, access frequency mean, and access time distribution sequence; the crawler risk type is determined based on the access frequency standard deviation, access frequency mean, and access time distribution sequence, and is used to characterize the preliminary judgment of the crawler result, and the crawler risk type includes low-frequency normal requests, high-frequency suspected crawler requests, concentrated period normal requests, uniformly distributed suspected crawler requests, and sudden peak requests; The step of constructing an access behavior feature matrix according to the updated access logs includes: Extract the access identifiers corresponding to all access behaviors in the updated access log, and obtain the access behavior parameters corresponding to all access behaviors in the updated access log; The access behavior feature matrix is ​​constructed by using access identifiers corresponding to all access behaviors in the updated access log as matrix rows and access behavior parameters corresponding to all access behaviors in the updated access log as matrix columns; A prediction module is used to input the access behavior parameters and the access feature parameters into a preset crawler recognition model to obtain a crawler recognition result corresponding to the current access behavior output by the preset crawler recognition model; the preset crawler recognition model is obtained by model training based on sample access behavior parameters and sample access feature parameters corresponding to multiple historical access behaviors and their sample crawler results; A processing module is used to perform crawler countermeasure on the current access behavior when the current access behavior corresponding to the crawler identification result is a malicious crawler behavior.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 5 is implemented.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images