Anti-crawler method and device based on machine learning, medium, equipment and product

Abstract

The invention relates to the technical field of computers, and provides an anti-crawler method and device based on machine learning, a medium, equipment and a product, and the method comprises the steps: obtaining real-time access traffic; based on a trained traffic identification model, identifying the real-time access traffic to obtain an identification result; and if the identification result represents that the real-time access traffic is the abnormal crawler behavior, defending the real-time access traffic. According to the anti-crawler method and device based on machine learning, the medium, the equipment and the product provided by the invention, the traffic identification model is constructed and trained, so that the real-time access traffic is intelligently identified, and normal user access and malicious crawler behaviors are accurately distinguished. And when the abnormal crawler behavior is identified, a defense mechanism is triggered in time, so that the security and availability of website contents are ensured. The accuracy of identifying abnormal crawler behaviors can be improved, so that the website data security is effectively protected, and the user experience is improved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to an anti-crawler method, device, medium, equipment and product based on machine learning. Background Art

[0002] With the rapid development of Internet technology, web crawlers, as an automated data capture tool, while promoting information aggregation and sharing, also bring security risks such as data leakage and excessive server load.

[0003] Therefore, anti-crawler technology is facing increasingly severe challenges. At present, most anti-crawler strategies rely on analyzing network request characteristics such as Uniform Resource Locator (URL) path, User-Agent, Headers, Cookies, and traffic statistics to formulate protection rules. Although this method can resist crawler activities to a certain extent, with the continuous evolution and innovation of crawler technology, traditional anti-crawler methods are often difficult to quickly adapt to new situations, resulting in a significant reduction in protection effects and an inability to form an effective long-term defense mechanism. Summary of the invention

[0004] The present invention provides an anti-crawler method, device, medium, equipment and product based on machine learning, which is used to solve the defect that the anti-crawler means in the prior art are difficult to quickly adapt to new situations, resulting in poor protection effect.

[0005] The present invention provides an anti-crawler method based on machine learning, comprising: Get real-time access traffic; Based on the trained traffic identification model, the real-time access traffic is identified to obtain an identification result; If the identification result indicates that the real-time access traffic is abnormal crawler behavior, defending the real-time access traffic; The step of acquiring the traffic identification model includes: Obtain access request samples and their corresponding labels; Extracting features from the access request sample to obtain access behavior features; Based on the access behavior characteristics and the labels, a pre-built initial model is trained to obtain the traffic identification model.

[0006] According to the anti-crawler method based on machine learning provided by the present invention, obtaining access request samples includes: Get a sample of original access requests; The original access request sample is preprocessed to obtain the access request sample.

[0007] According to the anti-crawler method based on machine learning provided by the present invention, the access request sample includes historical request data and user submitted data, and the feature extraction of the access request sample to obtain access behavior features includes: Extracting features from the historical request data and the user submitted data respectively, to obtain request data features and user data features respectively; The access behavior characteristics are determined based on the request data characteristics and the user data characteristics.

[0008] According to the anti-crawler method based on machine learning provided by the present invention, after training the pre-built initial model based on the access behavior characteristics and the labels to obtain the traffic identification model, the method further includes: regularly updating the access request sample; The traffic identification model is retrained based on the updated access request samples.

[0009] The present invention also provides an anti-crawler device based on machine learning, comprising: A traffic acquisition unit, used to acquire real-time access traffic; An identification unit, configured to identify the real-time access traffic based on a trained traffic identification model to obtain an identification result; A defense unit, configured to defend the real-time access traffic if the identification result indicates that the real-time access traffic is abnormal crawler behavior; The step of acquiring the traffic identification model includes: Obtain access request samples and their corresponding labels; Extracting features from the access request sample to obtain access behavior features; Based on the access behavior characteristics and the labels, a pre-built initial model is trained to obtain the traffic identification model.

[0010] According to the anti-crawler device based on machine learning provided by the present invention, the device also includes a model training unit, which is used to: Get a sample of original access requests; The original access request sample is preprocessed to obtain the access request sample.

[0011] According to the anti-crawler device based on machine learning provided by the present invention, the model training unit is also used for: Extracting features from the historical request data and the user submitted data respectively, to obtain request data features and user data features respectively; The access behavior characteristics are determined based on the request data characteristics and the user data characteristics.

[0012] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, an anti-crawler method based on machine learning as described above is implemented.

[0013] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-described anti-crawler methods based on machine learning.

[0014] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements any of the above-mentioned anti-crawler methods based on machine learning.

[0015] The anti-crawler method, device, medium, equipment and product based on machine learning provided by the present invention can intelligently identify real-time access traffic and accurately distinguish normal user access from malicious crawler behavior by building and training a traffic identification model. When abnormal crawler behavior is identified, the defense mechanism is triggered in time to ensure the security and availability of website content. It can improve the accuracy of identifying abnormal crawler behavior, thereby effectively protecting website data security and improving user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0017] Figure 1 This is one of the flow charts of the anti-crawler method based on machine learning provided by the present invention.

[0018] Figure 2 This is the second flow chart of the anti-crawler method based on machine learning provided by the present invention.

[0019] Figure 3 It is a structural schematic diagram of the anti-crawler device based on machine learning provided by the present invention.

[0020] Figure 4 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0022] Currently, most anti-crawling strategies rely on analyzing network request features such as URL paths, User-Agents, Headers, Cookies, and traffic statistics to formulate protection rules. Although this method can resist crawler activities to a certain extent, with the continuous evolution and innovation of crawler technology, traditional anti-crawler methods are often difficult to quickly adapt to new situations, resulting in a significant reduction in protection effects and the inability to form an effective long-term defense mechanism.

[0023] To address this problem, an embodiment of the present invention provides an anti-crawler method based on machine learning. By building and training a traffic identification model, real-time access traffic can be intelligently identified to accurately distinguish normal user access from malicious crawler behavior. When abnormal crawler behavior is identified, the defense mechanism is triggered in a timely manner to ensure the security and availability of website content. The method provided by an embodiment of the present invention can improve the accuracy of identifying abnormal crawler behavior, thereby effectively protecting website data security and improving user experience.

[0024] The embodiments of the present invention can be applied to scenarios where anti-crawler is required, such as e-commerce platforms, financial platforms, academic research platforms, etc. The execution subject of the method can be an electronic device such as a terminal device, a computer, a server, a server cluster, or a specially designed anti-crawler device, or an anti-crawler device set in the electronic device, which can be implemented by software, hardware, or a combination of the two.

[0025] Figure 1 This is one of the flow charts of the anti-crawler method based on machine learning provided by the present invention, such as Figure 1 As shown, the method comprises the following steps: Step 110, obtaining real-time access traffic; Step 120, based on the trained traffic identification model, identify the real-time access traffic to obtain an identification result; Step 130: If the identification result indicates that the real-time access traffic is abnormal crawler behavior, then defense is performed on the real-time access traffic; Among them, the steps of obtaining the traffic identification model include: obtaining access request samples and their corresponding labels; extracting features from the access request samples to obtain access behavior features; and training a pre-built initial model based on the access behavior features and labels to obtain a traffic identification model.

[0026] Specifically, all access traffic to a website or application platform is captured in real time, including normal user access and possible crawler access.

[0027] Ways to obtain real-time access traffic include: using network monitoring tools (such as Wireshark, tcpdump, etc.) to capture network packets flowing through the server. Configuring the logging function of the web server (such as Apache, Nginx, etc.) to record all access requests. Using the log function of the load balancer or reverse proxy server to collect access traffic data.

[0028] The website backend deploys a trained traffic identification model, which is used to intelligently analyze real-time access traffic to distinguish between normal access and abnormal crawler behavior. The identification results obtained may include normal access or abnormal crawler behavior. The traffic identification model can be a trained machine learning model or a deep learning model.

[0029] In specific implementation, real-time access traffic data can be input into a trained traffic identification model. The model performs intelligent analysis based on the characteristics of the input data to determine whether each access request is a normal access or abnormal crawler behavior.

[0030] Furthermore, if the identification result indicates that the real-time access traffic is abnormal crawler behavior, defense is performed on the real-time access traffic. The purpose of the defense is to prevent the abnormal crawler behavior from infringing on the website or application platform.

[0031] In some embodiments, defending real-time access traffic may include: IP blocking: Add the identified abnormal crawler IP addresses to the blacklist and directly deny their access. Verification code verification: Require the verification code to be entered for access requests from suspected crawlers, increasing their access difficulty. Dynamic page technology: Use JavaScript, Ajax and other technologies to dynamically generate page content, making it difficult for crawlers to crawl. Frequency limitation: Limit the access frequency of the same IP address or user account to prevent high-frequency access. Log auditing: Record and analyze abnormal access behaviors to provide data support for subsequent defense measures.

[0032] Before executing step 120, the initial model may be trained to obtain a traffic identification model. The steps of obtaining the traffic identification model include: Obtain access request samples and their corresponding labels. Collect historical access request data, including normal user access and known crawler access. Label the collected data and label normal access and crawler access respectively. Access request samples include website access logs, user behavior data (such as access frequency, access path, dwell time, etc.), IP address information, etc.

[0033] Extract features from access request samples to obtain access behavior features. Access behavior features refer to indicators that can characterize user behavior features. For example, request header information (such as User-Agent, Referer, IP address, etc.) can be extracted. Analyze request behavior features (such as access frequency, access path, dwell time, etc.) and extract data packet content features (such as request parameters, response content, etc.).

[0034] Then, select an appropriate machine learning algorithm (such as decision tree, random forest, neural network, etc.) to build an initial model for classification. Use labeled access request samples and their features to train the model. Use cross-validation, confusion matrix, ROC curve and other methods to evaluate model performance to ensure that the model has a high accuracy rate and a low false alarm rate, thereby optimizing model performance.

[0035] The method provided by the embodiment of the present invention constructs and trains a traffic identification model to intelligently identify real-time access traffic and accurately distinguish normal user access from malicious crawler behavior. When abnormal crawler behavior is identified, the defense mechanism is triggered in a timely manner to ensure the security and availability of website content. The accuracy of identifying abnormal crawler behavior can be improved, thereby effectively protecting website data security and improving user experience.

[0036] Based on the above embodiment, obtaining an access request sample includes: Get a sample of original access requests; The original access request sample is preprocessed to obtain an access request sample.

[0037] Specifically, considering that the original access request sample may contain some noise, in order to improve the accuracy of model training, after obtaining the original access request sample, the original access request sample is preprocessed to obtain the access request sample. The preprocessing may include at least one of data cleaning, denoising, standardization, and extraction of key information.

[0038] For example, remove sensitive information: such as passwords, keys, personal information, etc. to ensure data security. Format the request header and request body to make them easier to read or analyze. Extract key information: such as IP address, user agent, request timestamp, etc. for subsequent analysis. Encoding conversion: If the request contains data that is not encoded in UTF-8, encoding conversion may be required.

[0039] Based on any of the above embodiments, the access request sample includes historical request data and user submitted data, and feature extraction is performed on the access request sample to obtain access behavior features, including: Perform feature extraction on historical request data and user submitted data respectively to obtain request data features and user data features respectively; Determine access behavior characteristics based on request data characteristics and user data characteristics.

[0040] Specifically, historical request data is obtained from historical logs, databases or special storage systems. These data usually include request timestamps, IP addresses, request methods (such as GET, POST), request URIs, request headers, request bodies, etc.

[0041] User submitted data usually refers to the data submitted to the server by the user through forms, API calls, etc. This data can be included in the request body, such as data packets in JSON, XML and other formats, or may be passed through URL parameters. User submitted data may include user identity information, operation instructions, business data, etc.

[0042] Feature extraction is performed on historical request data to obtain request data features, which may specifically include: Time characteristics: By extracting the request timestamp, we can further analyze the time distribution and frequency of access.

[0043] IP address characteristics: IP addresses are recorded and can be used to identify geographic locations and abnormal access patterns (such as access with frequently changing IP addresses).

[0044] Request method characteristics: Analyze the request type (GET, POST, etc.) to understand the data interaction method.

[0045] URI features: Parse the requested URI to understand the specific page or resource the user accesses.

[0046] Request header features: Extract request header information such as User-Agent, Referer, Accept, etc., analyze client type, source page, acceptable response type, etc.

[0047] Request body features: Parse the request body to extract key business data or instructions.

[0048] Extract features from the data submitted by the user to obtain user data features. User data features may include: User identity information: Extract user ID, user name and other identity information from the submitted data to identify the user.

[0049] Operation instructions: Extract the operation instructions requested by the user, such as query, update, delete, etc.

[0050] Business data: Analyze the submitted business data to understand its structure, type, content, etc. for subsequent processing.

[0051] Data integrity check: Verify the data to check whether there are missing, abnormal or illegal values.

[0052] Based on the extracted request data features and user data features, access behavior features can be further determined, including but not limited to: Access pattern: Identify the patterns of normal access and abnormal access (such as brute force cracking and crawler access) through time characteristics and request frequency analysis.

[0053] User behavior: Combine user identity information and operation instructions to analyze user behavior patterns, such as frequent visits to specific pages and performing a large number of similar operations.

[0054] Data interaction characteristics: Analyze the data interaction between the request body and the response body to understand the data flow, data format, etc.

[0055] Therefore, risk assessment can be performed based on access behavior characteristics: based on the above characteristics, risk assessment can be performed on access requests to determine whether there are security risks, such as SQL injection, cross-site scripting (XSS) and other attack risks.

[0056] Based on any of the above embodiments, after training the pre-built initial model based on the access behavior characteristics and labels to obtain the traffic identification model, the method further includes: Regularly update the access request sample; Retrain the traffic identification model based on the updated access request samples.

[0057] Specifically, in order to adapt the model to new crawler technologies and attack methods, the performance of the model in actual applications is continuously monitored, and new access request samples and feedback are collected. Based on the monitoring results and feedback information, the training data is regularly updated and the model is retrained to maintain the accuracy and adaptability of the model.

[0058] Set up scheduled tasks or triggers to automatically capture the latest access request data from server logs, databases, or API interfaces. Ensure that the collected data comes from multiple sources to cover a wider range of access scenarios and user behaviors, so that access request samples can be updated regularly.

[0059] Based on the updated access request samples, the traffic identification model is retrained. The retraining process here can refer to the training process of the initial model, which will not be repeated here.

[0060] Based on any of the above embodiments, Figure 2 This is the second flow chart of the anti-crawler method based on machine learning provided by the present invention, such as Figure 2 As shown, the method includes: 1. Data collection and preprocessing

[0061] Collect website access logs, user behavior data (such as access frequency, access path, dwell time, etc.), IP address information, etc.

[0062] The collected data is cleaned, denoised and standardized to ensure data quality.

[0063] 2. Feature extraction

[0064] Based on historical data, extract indicators that can characterize user behavior characteristics, such as abnormal access frequency, missing request header information, abnormal access time distribution, etc.

[0065] Use text mining, image processing and other technologies to extract additional features from user-submitted form data, verification code input, etc.

[0066] 3. Machine Learning Model Construction

[0067] Select a suitable machine learning algorithm (such as random forest, gradient boosted tree, neural network, etc.) and build a classification model based on the extracted features.

[0068] Use labeled data sets (including normal access and malicious crawler access samples) to train the model and adjust model parameters to optimize classification performance.

[0069] 4. Model evaluation and optimization

[0070] Cross-validation, confusion matrix, ROC curve and other methods are used to evaluate model performance to ensure that the model has high accuracy and low false alarm rate.

[0071] Based on the evaluation results, the model is iteratively optimized to improve the recognition effect.

[0072] 5. Real-time monitoring and defense

[0073] Deploy the trained model to the website backend to monitor real-time access traffic.

[0074] When suspected malicious crawler behavior is detected, the system automatically triggers defense mechanisms, such as adding verification code verification, limiting access frequency, banning IP addresses, etc.

[0075] 6. Dynamic update and adaptation

[0076] Establish a feedback mechanism to collect cases of misjudgment or omission of the model for continuous optimization of the model.

[0077] Regularly retrain models to adapt to new crawler techniques and attack methods.

[0078] When collecting, processing, and storing access request samples, relevant laws, regulations, and privacy policies must be strictly followed to ensure the security and privacy of user data. When selecting a model, the interpretability of the model must be considered so that the reasons and logic behind the model's decisions can be explained. During model training and deployment, computing and storage resources must be reasonably optimized to improve efficiency and reduce costs.

[0079] The method provided by the embodiment of the present invention can effectively prevent malicious crawlers from illegally capturing and abusing website data, and protect the security and integrity of website content. It can reduce problems such as excessive server load and response delay caused by malicious crawlers, and improve user access experience. It can improve website protection capabilities through technological innovation and enhance the competitiveness of enterprises in the Internet field.

[0080] The anti-crawler device based on machine learning provided by the present invention is described below. The anti-crawler device based on machine learning described below and the anti-crawler method based on machine learning described above can refer to each other.

[0081] Based on any of the above embodiments, Figure 3 is a schematic diagram of the structure of the anti-crawler device based on machine learning provided by the present invention, such as Figure 3 As shown, the device comprises: The traffic acquisition unit 310 is used to acquire real-time access traffic; The identification unit 320 is used to identify the real-time access traffic based on the trained traffic identification model to obtain an identification result; A defense unit 330, configured to defend the real-time access traffic if the identification result indicates that the real-time access traffic is abnormal crawler behavior; The step of acquiring the traffic identification model includes: Obtain access request samples and their corresponding labels; Extracting features from the access request sample to obtain access behavior features; Based on the access behavior characteristics and the labels, a pre-built initial model is trained to obtain the traffic identification model.

[0082] The anti-crawler device based on machine learning provided by the embodiment of the present invention constructs and trains a traffic identification model to intelligently identify real-time access traffic and accurately distinguish normal user access from malicious crawler behavior. When abnormal crawler behavior is identified, the defense mechanism is triggered in time to ensure the security and availability of website content. It can improve the accuracy of identifying abnormal crawler behavior, thereby effectively protecting website data security and improving user experience.

[0083] Based on any of the above embodiments, the device further includes a model training unit, which is used to: Get a sample of original access requests; The original access request sample is preprocessed to obtain the access request sample.

[0084] Based on any of the above embodiments, the model training unit is further used for: Extracting features from the historical request data and the user submitted data respectively, to obtain request data features and user data features respectively; The access behavior characteristics are determined based on the request data characteristics and the user data characteristics.

[0085] Based on any of the above embodiments, the model training unit is further used for: regularly updating the access request sample; The traffic identification model is retrained based on the updated access request samples.

[0086] Figure 4 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 4 As shown, the electronic device may include: a processor 410, a communication interface 420, a memory 430 and a communication bus 440, wherein the processor 410, the communication interface 420 and the memory 430 communicate with each other through the communication bus 440. The processor 410 may call the logic instructions in the memory 430 to execute the anti-crawler method based on machine learning, the method comprising: obtaining real-time access traffic; based on the trained traffic identification model, identifying the real-time access traffic to obtain an identification result; if the identification result indicates that the real-time access traffic is an abnormal crawler behavior, then defending the real-time access traffic; wherein the acquisition step of the traffic identification model comprises: obtaining access request samples and their corresponding labels; extracting features from the access request samples to obtain access behavior features; based on the access behavior features and the labels, training a pre-built initial model to obtain the traffic identification model.

[0087] In addition, the logic instructions in the above-mentioned memory 430 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0088] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the anti-crawler method based on machine learning provided by the above methods, the method including: obtaining real-time access traffic; based on a trained traffic identification model, identifying the real-time access traffic to obtain an identification result; if the identification result characterizes that the real-time access traffic is abnormal crawler behavior, defending the real-time access traffic; wherein the step of obtaining the traffic identification model includes: obtaining access request samples and their corresponding labels; performing feature extraction on the access request samples to obtain access behavior features; based on the access behavior features and the labels, training a pre-built initial model to obtain the traffic identification model.

[0089] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the anti-crawler method based on machine learning provided by the above-mentioned methods, the method comprising: obtaining real-time access traffic; identifying the real-time access traffic based on a trained traffic identification model to obtain an identification result; if the identification result characterizes that the real-time access traffic is abnormal crawler behavior, defending the real-time access traffic; wherein the step of obtaining the traffic identification model comprises: obtaining access request samples and their corresponding labels; performing feature extraction on the access request samples to obtain access behavior features; and training a pre-built initial model based on the access behavior features and the labels to obtain the traffic identification model.

[0090] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0091] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0092] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An anti-crawler method based on machine learning, characterized in that: include: Get real-time access traffic; Based on the trained traffic identification model, the real-time access traffic is identified to obtain an identification result; If the identification result indicates that the real-time access traffic is abnormal crawler behavior, defending the real-time access traffic; The step of acquiring the traffic identification model includes: Obtain access request samples and their corresponding labels; Extracting features from the access request sample to obtain access behavior features; Based on the access behavior characteristics and the labels, a pre-built initial model is trained to obtain the traffic identification model.

2. The anti-crawler method based on machine learning according to claim 1, characterized in that: The obtaining of the access request sample includes: Get a sample of original access requests; The original access request sample is preprocessed to obtain the access request sample.

3. The anti-crawler method based on machine learning according to claim 1, characterized in that: The access request sample includes historical request data and user submitted data. The feature extraction of the access request sample to obtain access behavior features includes: Extracting features from the historical request data and the user submitted data respectively, to obtain request data features and user data features respectively; The access behavior characteristics are determined based on the request data characteristics and the user data characteristics.

4. The anti-crawler method based on machine learning according to claim 1, characterized in that: After training the pre-built initial model based on the access behavior characteristics and the label to obtain the traffic identification model, the method further includes: regularly updating the access request sample; The traffic identification model is retrained based on the updated access request samples.

5. An anti-crawler device based on machine learning, characterized in that: include: A traffic acquisition unit, used to acquire real-time access traffic; An identification unit, configured to identify the real-time access traffic based on a trained traffic identification model to obtain an identification result; A defense unit, configured to defend the real-time access traffic if the identification result indicates that the real-time access traffic is abnormal crawler behavior; The step of acquiring the traffic identification model includes: Obtain access request samples and their corresponding labels; Extracting features from the access request sample to obtain access behavior features; Based on the access behavior characteristics and the labels, a pre-built initial model is trained to obtain the traffic identification model.

6. The anti-crawler device based on machine learning according to claim 5, characterized in that: The device also includes a model training unit, which is used to: Get a sample of original access requests; The original access request sample is preprocessed to obtain the access request sample.

7. The anti-crawler method based on machine learning according to claim 6, characterized in that: The model training unit is further used for: Extracting features from the historical request data and the user submitted data respectively, to obtain request data features and user data features respectively; The access behavior characteristics are determined based on the request data characteristics and the user data characteristics.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the anti-crawler method based on machine learning as described in any one of claims 1 to 4 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the anti-crawler method based on machine learning as described in any one of claims 1 to 4 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the anti-crawler method based on machine learning as described in any one of claims 1 to 4 is implemented.

Images