Intelligent crawler identification method and system based on deep learning

Abstract

The invention discloses an intelligent crawler identification method and system based on deep learning, and belongs to the technical field of computers and artificial intelligence, and the method comprises the steps of data collection and preprocessing, deep learning model selection and training, real-time monitoring and detection, and system integration and performance optimization. Data collection and preprocessing: collecting data and carrying out cleaning, noise reduction, feature extraction and feature conversion on the collected data; selecting and training a deep learning model, and selecting different models or a hybrid model according to the type of the data; and performing real-time monitoring and detection, deploying the trained model in an actual network environment, performing real-time monitoring and detection, monitoring an IP address of an access request, and particularly indicating crawler actions if a large number of requests from the same IP address or an IP address range are found. According to the method, normal users and crawlers can be distinguished, websites and service providers are helped to effectively prevent malicious crawlers, and data and resources of the websites and the service providers are protected against adverse effects.

Description

Technical Field

[0001] The present invention relates to the field of computer and artificial intelligence technology, and specifically to an intelligent crawler recognition method and system based on deep learning. Background Art

[0002] Malicious crawlers are a common problem in the current Internet environment. Common crawler behaviors include data collection, information theft, website content theft, etc. Malicious crawlers pose a potential threat to data, resources and user privacy, affecting the security and legitimate rights and interests of websites and online services. Summary of the invention

[0003] The technical task of the present invention is to address the above shortcomings and provide a deep learning-based intelligent crawler identification method and system that can distinguish normal users from crawlers, help websites and service providers effectively prevent malicious crawlers, and protect their data and resources from adverse effects.

[0004] The technical solution adopted by the present invention to solve its technical problem is:

[0005] A deep learning-based intelligent crawler identification method, the implementation of which includes data collection and preprocessing, deep learning model selection and training, real-time monitoring and detection, system integration and performance optimization;

[0006] The data collection and preprocessing includes collecting data and cleaning and denoising the collected data, extracting features and converting features;

[0007] The deep learning model selection and training selects different models or hybrid models according to the type of data;

[0008] The real-time monitoring and detection is to perform real-time monitoring and detection after the trained model is deployed in the actual network environment, monitoring the IP addresses of access requests, especially if a large number of requests from the same IP address or IP address range are found, which indicates crawler activities;

[0009] The system integration and performance optimization uses a distributed system architecture to process large-scale data and high-concurrency requests, and improves the scalability and performance of the system through horizontal expansion and load balancing.

[0010] Furthermore, the data collection and preprocessing collects network traffic data, including access logs of websites or services, data captured by network packet capture tools, or data collected by specially deployed data collectors; the data includes information such as the timestamp of the access request, the user agent string, the request path, and the IP address;

[0011] After data collection, preprocessing steps are performed, including:

[0012] Data cleaning and denoising: remove invalid or abnormal data points to ensure data consistency and reliability; data cleaning includes missing value processing, abnormal value processing, duplicate data processing, and data format unification; data denoising includes the use of smoothing techniques (such as moving average, exponential average, etc.) to reduce noise and fluctuations in the data, thereby making the data more stable and predictable;

[0013] Feature extraction: After preprocessing the data, useful features are extracted, such as request frequency, structure of request path, information in request header, etc., to distinguish different types of user behaviors; specific features are directly extracted from the raw data using predefined rules and logic; for example, path information is extracted from the URL, user agent string is extracted from the request header, etc.

[0014] Feature conversion and standardization: Perform appropriate conversion (such as one-hot encoding) and standardization (such as normalization or standardization) on features to facilitate subsequent model processing and training; perform one-hot encoding or label encoding on categorical features to convert them into numerical forms that can be processed by the model; standardize or normalize numerical features to keep their numerical ranges within a similar range to avoid the impact of scale differences between features on the model.

[0015] Furthermore, if the data has a time series nature, time-related features are extracted, including time intervals, distribution of time points, etc., to reveal the temporal pattern of user behavior;

[0016] For text data, natural language processing (NLP) technology is used, including bag of words model (Bag of Words), word embedding (Word Embedding) and other methods to extract the semantic features of the text.

[0017] Further, the identification of crawler behavior includes: crawlers have high-frequency request patterns, in contrast, normal user requests are usually more stable and random.

[0018] Furthermore, the deep learning model selection and training, the selected model includes:

[0019] Convolutional Neural Network (CNN): used for image recognition or for processing network request data with spatial structure;

[0020] Recurrent Neural Network (RNN) or Long Short-Term Memory (LSTM): used to process data with time series characteristics, including the time interval between user requests or the order of request paths;

[0021] The model training phase includes the following key steps:

[0022] Dataset preparation: Divide the dataset into training, validation, and test sets, and ensure that the dataset contains representative samples of normal users and malicious crawlers;

[0023] Model training: Use the training set to train the selected deep learning model and optimize the model parameters to maximize accuracy and generalization ability;

[0024] Model tuning: Optimize the model by adjusting hyperparameters, increasing data samples, or introducing regularization techniques to improve its ability to recognize different types of crawler behaviors.

[0025] Furthermore, the real-time monitoring and detection includes:

[0026] Data stream processing: real-time acquisition and processing of network traffic data, feature extraction and prediction for each request;

[0027] Anomaly detection: Use the trained deep learning model to classify requests and determine whether they are normal user behavior or malicious crawler behavior;

[0028] Real-time response: After identifying malicious crawler behavior, pre-set response measures are immediately taken, including blocking IP addresses, triggering verification code verification, or recording logs.

[0029] Furthermore, the system integration and performance optimization include:

[0030] System integration: Integrate the intelligent crawler identification system into the existing network security infrastructure to ensure that it is coordinated with other security measures;

[0031] Performance optimization: optimize the system's processing speed and resource utilization efficiency to ensure effective operation under high load and complex environments;

[0032] Continuous improvement: Monitor the system’s performance metrics and regularly update deep learning models to adapt to new crawler behaviors and network threats.

[0033] The present invention also claims protection for an intelligent crawler identification system based on deep learning, comprising:

[0034] Data collection and preprocessing module, used to collect data and clean and reduce noise, extract features and convert features of the collected data;

[0035] Deep learning model selection and training module, used to select different models or hybrid models according to the type of data;

[0036] The real-time monitoring and detection module is used to perform real-time monitoring and detection after the trained model is deployed in the actual network environment, monitor the IP addresses of access requests, and identify crawler actions;

[0037] System integration and performance optimization module, which uses distributed system architecture to handle large-scale data and high-concurrency requests, and improves system scalability and performance through horizontal expansion and load balancing;

[0038] The system performs crawler identification through the above-mentioned deep learning-based intelligent crawler identification method.

[0039] The present invention also claims protection for an intelligent crawler identification device based on deep learning, comprising: at least one memory and at least one processor;

[0040] The at least one memory is used to store a machine-readable program;

[0041] The at least one processor is used to call the machine-readable program to implement the above method.

[0042] The present invention also claims protection for a computer-readable medium having computer instructions stored thereon, which, when executed by a processor, causes the processor to perform the above method.

[0043] Compared with the prior art, the intelligent crawler identification method and system based on deep learning of the present invention have the following beneficial effects:

[0044] High-efficiency recognition rate: Deep learning technology is used to accurately identify complex crawler behaviors, greatly improving recognition accuracy and countermeasure effects.

[0045] Adaptive capability: The system can dynamically adjust model parameters and feature extraction strategies to meet the challenges of continuous evolution and improvement of crawler technology.

[0046] Privacy protection: The design takes user privacy protection into consideration, only collects minimal data related to identifying crawler behavior, and complies with data protection regulations.

[0047] The intelligent crawler identification method system based on deep learning not only improves the security protection level of websites, but also provides effective guarantee for the stable operation and user experience of Internet information services. This technology has important application prospects and market potential in the era of big data. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 It is a schematic diagram showing the principle of an intelligent crawler identification method based on deep learning provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0049] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments.

[0050] The embodiment of the present invention provides a method for intelligent crawler identification based on deep learning, the implementation of which includes data collection and preprocessing, deep learning model selection and training, real-time monitoring and detection, system integration and performance optimization. Crawlers have a high-frequency request pattern, while normal user requests are usually more stable and random.

[0051] The data collection and preprocessing includes collecting data and cleaning and denoising the collected data, extracting features and converting features to make the data more real and effective; and then performing deep learning model selection and training.

[0052] The deep learning model selection and training selects different models or a hybrid model according to the type of data.

[0053] The real-time monitoring and detection is performed after the trained model is deployed in the actual network environment to monitor the IP addresses of access requests. In particular, if a large number of requests are found from the same IP address or IP address range, it indicates crawler activity.

[0054] The system integration and performance optimization uses a distributed system architecture to process large-scale data and high-concurrency requests, and improves the scalability and performance of the system through horizontal expansion and load balancing.

[0055] Through the training of deep neural network models with large-scale data sets, automatic classification and identification of user behavior and crawler behavior can be achieved. The model is based on convolutional neural network (CNN) and recurrent neural network (RNN), which can extract features from dynamic data and analyze patterns and features in network data streams to distinguish normal users from crawlers and identify non-human operations.

[0056] By collecting and analyzing the interaction data between users and websites, such as mouse movement trajectory, keyboard input speed, page dwell time and other behavioral characteristics, combined with time series data analysis technology, a behavioral feature vector is constructed to provide input for the deep learning model.

[0057] By applying deep learning models to real-time traffic, we can quickly determine whether a request comes from a legitimate user or a crawler. For requests identified as malicious crawlers, we can take immediate response measures, such as denial of service, captcha verification, etc.

[0058] Through continuous monitoring and feedback mechanisms, the deep learning model is continuously optimized to adapt to new crawler attacks and changes in user behavior, maintaining efficient and accurate recognition capabilities.

[0059] This method helps websites and service providers effectively guard against malicious crawlers and protect their data and resources from adverse effects.

[0060] The following is combined with Figure 1As shown, the specific implementation of this method is further described in detail.

[0061] 1. Data collection and preprocessing.

[0062] First, you need to collect a large amount of network traffic data, which can come from the access logs of websites or services, data captured by network packet capture tools, or through specially deployed data collectors. The data may contain information such as the timestamp of the access request, the user agent string, the request path, and the IP address.

[0063] After data collection, preprocessing steps are performed, including:

[0064] Data cleaning and denoising: Remove invalid or abnormal data points to ensure data consistency and reliability. Data cleaning includes missing value processing, abnormal value processing, duplicate data processing, and data format unification; data denoising includes using smoothing techniques (such as moving average, exponential average, etc.) to reduce noise and fluctuations in the data, making the data more stable and predictable.

[0065] Feature extraction: After preprocessing the data, useful features are extracted, such as request frequency, structure of request path, information in request header, etc. These features help distinguish different types of user behaviors. Specific features are directly extracted from the raw data using predefined rules and logic. For example, path information is extracted from the URL, user agent string is extracted from the request header, etc. If the data has a time series nature, time-related features can be extracted, such as time intervals, distribution of time points, etc., to reveal the temporal pattern of user behavior. For text data, natural language processing (NLP) techniques such as Bag of Words and Word Embedding are used to extract semantic features of the text.

[0066] Feature conversion and standardization: Perform appropriate conversion (such as one-hot encoding) and standardization (such as normalization or standardization) on features to facilitate subsequent model processing and training. Perform one-hot encoding or label encoding on categorical features to convert them into numerical forms that can be processed by the model; standardize or normalize numerical features to keep their numerical ranges within a similar range to avoid the impact of scale differences between features on the model.

[0067] 2. Deep learning model selection and training.

[0068] Choosing the right deep learning model architecture is crucial to the effectiveness of the system. In the intelligent crawler recognition system, commonly used models include:

[0069] Convolutional Neural Network (CNN): A classic model for image recognition, also suitable for processing network request data with spatial structure.

[0070] Recurrent Neural Network (RNN) or Long Short-Term Memory Network (LSTM): used to process data with time series characteristics, such as the time interval between user requests or the order of request paths.

[0071] The model training phase includes the following key steps:

[0072] Dataset preparation: Divide the dataset into training, validation, and test sets, and ensure that the dataset contains representative samples of normal users and malicious crawlers.

[0073] Model training: Use the training set to train the selected deep learning model and optimize the model parameters to maximize accuracy and generalization ability.

[0074] Model tuning: Optimize the model by adjusting hyperparameters, increasing data samples, or introducing regularization techniques to improve its ability to recognize different types of crawler behaviors.

[0075] 3. Real-time monitoring and detection.

[0076] Once the trained model is deployed in the actual network environment, real-time monitoring and detection can be performed:

[0077] Data stream processing: Acquire and process network traffic data in real time, and perform feature extraction and prediction on each request.

[0078] Anomaly detection: Use the trained deep learning model to classify requests and determine whether they are normal user behavior or malicious crawler behavior.

[0079] Real-time response: After identifying malicious crawler behavior, pre-set response measures are immediately taken, including blocking IP addresses, triggering verification code verification, or recording logs.

[0080] 4. System integration and performance optimization.

[0081] Consider overall integration performance and stability, including:

[0082] System Integration: Integrate the intelligent crawler identification system into the existing network security infrastructure to ensure that it coordinates with other security measures.

[0083] Performance optimization: Optimize the system's processing speed and resource utilization efficiency to ensure effective operation under high load and complex environments.

[0084] Continuous improvement: Monitor the system’s performance metrics and regularly update deep learning models to adapt to new crawler behaviors and network threats.

[0085] An embodiment of the present invention also provides an intelligent crawler identification system based on deep learning, which performs crawler identification through the intelligent crawler identification method based on deep learning described in the above embodiment.

[0086] The system includes:

[0087] 1. Data collection and preprocessing module, used to collect data and clean and denoise the collected data, extract features and convert features.

[0088] First, collect a large amount of network traffic data, which can come from the access logs of websites or services, data captured by network packet capture tools, or through specially deployed data collectors. The data may contain information such as the timestamp of the access request, user agent string, request path, IP address, etc.

[0089] After data collection, preprocessing steps are performed, including:

[0090] Data cleaning and denoising: Remove invalid or abnormal data points to ensure data consistency and reliability. Data cleaning includes missing value processing, abnormal value processing, duplicate data processing, and data format unification; data denoising includes using smoothing techniques (such as moving average, exponential average, etc.) to reduce noise and fluctuations in the data, making the data more stable and predictable.

[0091] Feature extraction: After preprocessing the data, useful features are extracted, such as request frequency, structure of request path, information in request header, etc. These features help distinguish different types of user behaviors. Specific features are directly extracted from the raw data using predefined rules and logic. For example, path information is extracted from the URL, user agent string is extracted from the request header, etc. If the data has a time series nature, time-related features can be extracted, such as time intervals, distribution of time points, etc., to reveal the temporal pattern of user behavior. For text data, natural language processing (NLP) techniques such as Bag of Words and Word Embedding are used to extract semantic features of the text.

[0092] Feature conversion and standardization: Perform appropriate conversion (such as one-hot encoding) and standardization (such as normalization or standardization) on features to facilitate subsequent model processing and training. Perform one-hot encoding or label encoding on categorical features to convert them into numerical forms that can be processed by the model; standardize or normalize numerical features to keep their numerical ranges within a similar range to avoid the impact of scale differences between features on the model.

[0093] 2. Deep learning model selection and training module, used to select different models or hybrid models according to the type of data. Commonly used models include:

[0094] Convolutional Neural Network (CNN): A classic model for image recognition, also suitable for processing network request data with spatial structure.

[0095] Recurrent Neural Network (RNN) or Long Short-Term Memory Network (LSTM): used to process data with time series characteristics, such as the time interval between user requests or the order of request paths.

[0096] The model training phase involves the following key steps:

[0097] Dataset preparation: Divide the dataset into training, validation, and test sets, and ensure that the dataset contains representative samples of normal users and malicious crawlers.

[0098] Model training: Use the training set to train the selected deep learning model and optimize the model parameters to maximize accuracy and generalization ability.

[0099] Model tuning: Optimize the model by adjusting hyperparameters, increasing data samples, or introducing regularization techniques to improve its ability to recognize different types of crawler behaviors.

[0100] 3. Real-time monitoring and detection module, which is used to perform real-time monitoring and detection after the trained model is deployed in the actual network environment, monitor the IP address of the access request, and identify crawler actions. Crawlers have a high-frequency request pattern. In contrast, normal user requests are usually more stable and random.

[0101] Once the trained model is deployed in the actual network environment, the system can perform real-time monitoring and detection:

[0102] Data stream processing: The system acquires and processes network traffic data in real time, extracts features and makes predictions for each request.

[0103] Anomaly detection: Use the trained deep learning model to classify requests and determine whether they are normal user behavior or malicious crawler behavior.

[0104] Real-time response: After identifying malicious crawler behavior, the system can immediately take pre-set response measures, such as blocking IP addresses, triggering verification code verification, or recording logs.

[0105] 4. System integration and performance optimization module, which uses distributed system architecture to process large-scale data and high-concurrency requests, and improves the scalability and performance of the system through horizontal expansion and load balancing.

[0106] Finally, the system needs to take into account overall integration performance and stability, including:

[0107] System Integration: Integrate the intelligent crawler identification system into the existing network security infrastructure to ensure that it is coordinated with other security measures.

[0108] Performance optimization: Optimize the system's processing speed and resource utilization efficiency to ensure effective operation under high load and complex environments.

[0109] Continuous improvement: Monitor the system’s performance metrics and regularly update deep learning models to adapt to new crawler behaviors and network threats.

[0110] An embodiment of the present invention further provides an intelligent crawler identification device based on deep learning, comprising: at least one memory and at least one processor;

[0111] The at least one memory is used to store a machine-readable program;

[0112] The at least one processor is used to call the machine-readable program to implement the deep learning-based intelligent crawler identification method described in the above embodiment.

[0113] The embodiment of the present invention further provides a computer-readable medium, on which computer instructions are stored, and when the computer instructions are executed by a processor, the processor executes the intelligent crawler identification method based on deep learning described in the above embodiment. Specifically, a system or device equipped with a storage medium can be provided, on which a software program code that implements the functions of any of the above embodiments is stored, and a computer (or CPU or MPU) of the system or device reads and executes the program code stored in the storage medium.

[0114] In this case, the program code itself read from the storage medium can realize the function of any one of the above-mentioned embodiments, and thus the program code and the storage medium storing the program code constitute a part of the present invention.

[0115] The storage medium embodiments for providing the program code include a floppy disk, a hard disk, a magneto-optical disk, an optical disk (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD+RW), a magnetic tape, a non-volatile memory card, and a ROM. Alternatively, the program code can be downloaded from a server computer by a communication network.

[0116] In addition, it should be clear that the functions of any of the above embodiments can be implemented not only by executing the program code read by the computer, but also by enabling an operating system operating on the computer to complete part or all of the actual operations based on instructions from the program code.

[0117] In addition, it can be understood that the program code read from the storage medium is written to a memory provided in an expansion board inserted into the computer or written to a memory provided in an expansion unit connected to the computer, and then based on the instructions of the program code, a CPU installed on the expansion board or the expansion unit is enabled to perform part or all of the actual operations, thereby realizing the functions of any of the above-mentioned embodiments.

[0118] The present invention is shown and described in detail above through the accompanying drawings and preferred embodiments. However, the present invention is not limited to these disclosed embodiments. Based on the above multiple embodiments, those skilled in the art can know that the code review methods in the above different embodiments can be combined to obtain more embodiments of the present invention, and these embodiments are also within the protection scope of the present invention.

Claims

1. A deep learning-based intelligent crawler identification method, characterized in that: The implementation of this method includes data collection and preprocessing, deep learning model selection and training, real-time monitoring and detection, system integration and performance optimization; The data collection and preprocessing includes collecting data and cleaning and denoising the collected data, extracting features and converting features; The deep learning model selection and training selects different models or hybrid models according to the type of data; The real-time monitoring and detection is to perform real-time monitoring and detection after the trained model is deployed in the actual network environment, monitoring the IP addresses of access requests, especially if a large number of requests from the same IP address or IP address range are found, which indicates crawler activities; The system integration and performance optimization uses a distributed system architecture to process large-scale data and high-concurrency requests, and improves the scalability and performance of the system through horizontal expansion and load balancing.

2. According to the deep learning-based intelligent crawler identification method of claim 1, it is characterized in that: The data collection and preprocessing includes collecting network traffic data, including access logs of websites or services, data captured by network packet capture tools, or data collected by specially deployed data collectors; The data includes the timestamp of the access request, the user agent string, the request path, and the IP address information; After data collection, preprocessing steps are performed, including: Data cleaning and denoising: remove invalid or abnormal data points to ensure data consistency and reliability; data cleaning includes missing value processing, abnormal value processing, duplicate data processing, and data format unification; data denoising includes using smoothing technology to reduce noise and fluctuations in data to make data more stable and predictable; Feature extraction: After preprocessing the data, useful features are extracted to distinguish different types of user behaviors; specific features are directly extracted from the raw data using predefined rules and logic; Feature conversion and standardization: Convert and standardize features to facilitate subsequent model processing and training; perform one-hot encoding or label encoding on categorical features to convert them into numerical forms that can be processed by the model; standardize or normalize numerical features to keep their numerical range within a similar range.

3. The method for intelligent crawler identification based on deep learning according to claim 2, characterized in that: The feature extraction, if the data has a time series nature, extracts time-related features, including time intervals and distribution of time points, to reveal the time pattern of user behavior; For text data, natural language processing techniques, including bag-of-words model and word embedding method, are used to extract semantic features of text.

4. The method for intelligent crawler identification based on deep learning according to claim 3, characterized in that: Identification of crawler behavior includes: crawlers have high-frequency request patterns, in contrast, normal users' requests are more stable and random.

5. The method for identifying intelligent crawlers based on deep learning according to claim 1 or 4, characterized in that: The deep learning model selection and training, the selected models include: Convolutional neural networks: used for image recognition or for processing network request data with spatial structure; Recurrent neural networks or long short-term memory networks: used to process data with time series characteristics, including the time interval between user requests or the order of request paths; The model training phase includes the following key steps: Dataset preparation: Divide the dataset into training, validation, and test sets, and ensure that the dataset contains representative samples of normal users and malicious crawlers; Model training: Use the training set to train the selected deep learning model and optimize the model parameters to maximize accuracy and generalization ability; Model tuning: Optimize the model by adjusting hyperparameters, increasing data samples, or introducing regularization techniques to improve its ability to recognize different types of crawler behaviors.

6. The method for identifying intelligent crawlers based on deep learning according to claim 1 or 4, characterized in that: The real-time monitoring and detection includes: Data stream processing: real-time acquisition and processing of network traffic data, feature extraction and prediction for each request; Anomaly detection: Use the trained deep learning model to classify requests and determine whether they are normal user behavior or malicious crawler behavior; Real-time response: After identifying malicious crawler behavior, pre-set response measures are immediately taken, including blocking IP addresses, triggering verification code verification, or recording logs.

7. The method for intelligent crawler identification based on deep learning according to claim 1, characterized in that: The system integration and performance optimization include: System integration: Integrate the intelligent crawler identification system into the existing network security infrastructure to ensure that it is coordinated with other security measures; Performance optimization: optimize the system's processing speed and resource utilization efficiency to ensure effective operation under high load and complex environments; Continuous improvement: Monitor the system’s performance metrics and regularly update deep learning models to adapt to new crawler behaviors and network threats.

8. An intelligent crawler recognition system based on deep learning, characterized in that: include: Data collection and preprocessing module, used to collect data and clean and reduce noise, extract features and convert features of the collected data; Deep learning model selection and training module, used to select different models or hybrid models according to the type of data; The real-time monitoring and detection module is used to perform real-time monitoring and detection after the trained model is deployed in the actual network environment, monitor the IP addresses of access requests, and identify crawler actions; System integration and performance optimization module, which uses distributed system architecture to handle large-scale data and high-concurrency requests, and improves system scalability and performance through horizontal expansion and load balancing; The system performs crawler identification by using the deep learning-based intelligent crawler identification method described in any one of claims 1 to 7.

9. An intelligent crawler identification device based on deep learning, characterized in that: include: at least one memory and at least one processor; The at least one memory is used to store a machine-readable program; The at least one processor is used to call the machine-readable program to implement the method described in any one of claims 1 to 7.

10. A computer-readable medium, characterized in that The computer readable medium stores computer instructions, which, when executed by a processor, cause the processor to execute the method according to any one of claims 1 to 7.

Images