A crawler protection system, method and computer readable storage medium

Abstract

The present disclosure relates to the technical field of network security, and particularly relates to a crawler protection system and method and a computer readable storage medium, the system comprising: a crawler protection intelligent agent and a crawler protection model based on LSTM; wherein the crawler protection intelligent agent is configured to: transmit access request information to the crawler protection model; receive analysis information of the access request information from the crawler protection model, determine a malicious value of the access request information; and execute a crawler protection strategy corresponding to the malicious value. The present disclosure makes full use of real-time access sequence features, time features and behavior statistical features, and provides data storage and data management, data preprocessing and feature extraction, real-time data updating and the like for intelligent crawler protection model training through efficient design of data storage, and designs efficient algorithms based on efficient data management modes, so as to identify abnormalities in real time and take preventive measures in a timely manner.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of network security, and particularly relates to a crawler protection system and method and a computer readable storage medium. BACKGROUND

[0002] With the advent of the digital and networked era, information security issues have gradually entered the field of vision of people. With the rapid development of crawler technology, as a data collection tool, web crawlers often cause the leakage of website information and personal information by collecting a large amount of background data of websites. Especially the activities of malicious crawlers have posed a serious threat to the information security in the civil aviation field. The information security in the civil aviation field is facing unprecedented challenges.

[0003] Traditional crawler protection technologies are often based on rules, such as IP blocking, User-Agent verification, Cookie verification, etc. The main means are to block suspicious IPs by using black and white lists, to judge disguised request information through request header information, and to check whether the value of Cookie is abnormal. Although these methods are simple and direct to implement, they can block specific IP crawler requests or poorly disguised crawler requests. However, these strategies have the risk of misjudging normal users, which may affect the experience of normal users. In serious cases, it may lead to the loss of users of airlines, resulting in loss of revenue. Moreover, with the continuous iteration and development of web crawler technology, the traditional rule-based crawler protection mode cannot meet the requirements of ensuring information security, and the traditional strategies also need to be continuously updated and iterated. SUMMARY

[0004] The present disclosure aims to at least partially solve one of the above technical problems, and for this purpose, a crawler protection system is proposed, comprising:

[0005] a crawler protection intelligent agent and a crawler protection model based on LSTM; wherein,

[0006] the crawler protection intelligent agent is configured to:

[0007] transmit the access request information to the crawler protection model;

[0008] receive the analysis information of the access request information from the crawler protection model, and determine a malicious value of the access request information;

[0009] execute a crawler protection strategy corresponding to the malicious value.

[0010] Further, the crawler protection intelligent agent comprises:

[0011] a crawler protection business end and a crawler protection service end; wherein,

[0012] The crawler protection service end is used for performing service end crawler protection according to a service end expert protection strategy; wherein the service end crawler protection comprises:

[0013] The white list, request release, difficulty improvement, periodic prohibition and permanent prohibition;

[0014] The crawler protection service end is used for performing service end crawler protection according to a service end expert protection strategy; wherein the service end crawler protection comprises:

[0015] Scoring the access request based on the service end expert protection strategy.

[0016] Further, the service end crawler protection further comprises:

[0017] Sending the access request information corresponding to the access request to the crawler protection model; and

[0018] Receiving the access request score value returned by the crawler protection model.

[0019] Further, the service end crawler protection further comprises:

[0020] According to the score value of the access request, the access request is processed correspondingly.

[0021] Further, the crawler protection intelligent agent is further used for performing crawler identification based on big data processing technology, and the method comprises:

[0022] Periodically grabbing log data corresponding to the access request;

[0023] Standardizing and cleaning the log data;

[0024] Aggregating and analyzing the processed data to determine whether the current access request is from a crawler.

[0025] Further, the training process of the crawler protection model comprises:

[0026] Collecting behavior characteristic data of the service end user;

[0027] Training the crawler protection model according to the behavior characteristic data; wherein the crawler protection model is composed of an LSTM layer, a neural network layer and a full connection layer connected in sequence; the LSTM layer comprises input gate, forget gate and output gate.

[0028] Constructing the crawler protection model based on the trained crawler protection model.

[0029] Further, the behavior characteristic data comprises:

[0030] Sequence features and behavior statistical features; wherein,

[0031] The sequence features include: access operation sequences and access page sequences.

[0032] Further, the LSTM layer of the crawler protection model contains two LSTM networks, respectively used for processing the access operation sequences and the access page sequences.

[0033] The neural network layer is used for processing the behavior statistical features.

[0034] The present disclosure also provides a crawler protection method based on the crawler protection system, comprising:

[0035] The crawler protection intelligent agent is used to:

[0036] The access request information is transmitted to the crawler protection model;

[0037] The analysis information of the crawler protection model on the access request information is received;

[0038] The crawler protection strategy corresponding to the analysis information is executed.

[0039] The present disclosure also provides a computer readable storage medium, which stores a computer program or instructions, and the computer program or instructions are executed by a processor to at least implement the method.

[0040] Compared with the prior art, the present disclosure has the following beneficial effects:

[0041] The present disclosure fully utilizes real-time access sequence features, time features, and behavior statistical features, and provides data storage and data management, data preprocessing and feature extraction, real-time data updating, and the like for intelligent crawler protection model training based on efficient data management mode, and designs efficient algorithms to identify user requests in real time and take preventive measures in a timely manner.

[0042] Other features and advantages of the present disclosure will be described in the following description, and some will become apparent from the description, or will be understood from the practice of the present disclosure. The purposes and other advantages of the present disclosure can be achieved and obtained by the structures specifically pointed out in the written description and the accompanying drawings. The technical solutions of the present disclosure will be further described below by the accompanying drawings and examples. BRIEF DESCRIPTION OF DRAWINGS

[0043] The accompanying drawings are used to provide a further understanding of the present disclosure, and constitute a part of the specification, and are used to explain the present disclosure together with the embodiments of the present disclosure, and do not constitute a limitation to the present disclosure. In the drawings:

[0044] Figure 1 a schematic diagram of a crawler protection system given for the present application;

[0045] Figure 2 a schematic diagram of a crawler protection system given for the present application;

[0046] Figure 3 a training flowchart of a crawler protection model given for the present application;

[0047] Figure 4 a structural schematic diagram of a crawler protection model given for the present application;

[0048] Figure 5 a schematic diagram of a data processing flow of a crawler protection model given for the present application;

[0049] Figure 6 a running flowchart of a crawler protection model given for the present application;

[0050] Figure 7 a schematic diagram of an electronic device given for the present application;

[0051] Figure 8 a schematic diagram of a computer readable storage medium given for the present application. DETAILED DESCRIPTION

[0052] The present disclosure is described below in conjunction with the accompanying drawings, the preferred embodiments described herein are only used to illustrate and explain the present disclosure, and are not used to limit the present disclosure.

[0053] As shown in Figure 1 , the present disclosure gives a crawler protection system, comprising:

[0054] a crawler protection intelligent agent and a crawler protection model based on LSTM; wherein,

[0055] the crawler protection intelligent agent is used for:

[0056] transmitting access request information to the crawler protection model;

[0057] receiving analysis information of the crawler protection model on the access request information, and determining a malicious value of the access request information;

[0058] executing a crawler protection strategy corresponding to the malicious value.

[0059] Further, the crawler protection intelligent agent comprises:

[0060] a crawler protection business end and a crawler protection service end; wherein,

[0061] The crawler protection service end is configured to perform service end crawler protection according to a service end expert protection strategy; the service end crawler protection comprises: white list marking, request release, difficulty improvement, periodic prohibition and permanent prohibition.

[0062] The crawler protection service end is configured to perform service end crawler protection according to a service end expert protection strategy; the service end crawler protection comprises: white list marking, request release, difficulty improvement, periodic prohibition and permanent prohibition.

[0063] Further, the service end crawler protection further comprises: sending access request information corresponding to the access request to the crawler protection model; and receiving an access request score value returned by the crawler protection model.

[0064] Further, the service end crawler protection further comprises: performing corresponding processing on the access request according to the score value of the access request.

[0065] According to some embodiments of the present disclosure, as Figure 2 The intelligent crawler protection system comprises an intelligent crawler protection intelligent agent and a machine learning platform, wherein the machine learning platform is a model training platform for developing a crawler protection model, and the core is a crawler protection model based on LSTM. The intelligent crawler protection system business intelligent agent comprises an intelligent crawler protection business end and an intelligent crawler protection service end, the intelligent crawler protection business end performs five types of operations of white list marking, request release, difficulty improvement, periodic prohibition and permanent prohibition according to a relatively basic expert protection strategy in the client end. The intelligent crawler protection service end is composed of a service end expert strategy scoring and a machine learning platform scoring, and jointly intercepts malicious crawlers and malicious access requests. The machine learning platform stores user behavior feature data sets in a feature center for training of a machine learning model, the trained machine learning model provides services uniformly, and gives a risk score of a request in real time according to an input of the intelligent crawler protection system.

[0066] Further, the crawler protection intelligent agent is further configured to perform crawler identification based on a big data processing technology, comprising:

[0067] Periodically capturing log data corresponding to an access request;

[0068] Performing standardization processing and data cleaning processing on the log data;

[0069] Performing aggregation analysis on the processed data to determine whether the current access request is from a crawler.

[0070] According to some embodiments of the present disclosure, the big data-based crawler identification technology is as follows: first, the log data stored in es is real-time captured by Sqoop to ensure the timeliness and integrity of the data, and then the collected log is standardized and converted into a unified format and stored on the big data platform hdfs (distributed file system); second, the log table structure is built in the hive warehouse and stored in the original layer ods, the data is mapped to the ods table, the data is processed through data cleaning and data conversion, the log is desensitized, the null values, outliers, repeated logs and missing values in the data table are removed, and the data quality is improved; the processed data is stored in the dwd layer as an intermediate table, then the intermediate table is analyzed, the required fields are extracted, the required data is aggregated, the total number, average value, etc. are calculated, and finally the calculated results are stored in the ads layer for marking, and whether the current IP is a crawler is determined according to the calculated index.

[0071] Further, the training process of the crawler protection model comprises:

[0072] Collecting behavior feature data of business end users;

[0073] Training a crawler protection model according to the behavior feature data; wherein the crawler protection model is composed of an LSTM layer, a neural network layer and a full connection layer connected in sequence; the LSTM layer comprises input gate, forget gate and output gate control unit;

[0074] Building a crawler protection model based on the trained crawler protection model.

[0075] Further, the behavior feature data comprises sequence features and behavior statistical features; wherein the sequence features comprise access operation sequence and access page sequence.

[0076] Further, the LSTM layer of the crawler protection model contains two LSTM networks, respectively used for processing the access operation sequence and the access page sequence; the neural network layer is used for processing the behavior statistical features.

[0077] According to some embodiments of the present disclosure, Figure 3A machine learning platform intelligent crawler protection model training flowchart is provided. The machine learning platform is constructed based on a crawler protection algorithm. The construction process of the crawler protection algorithm is as follows: user behavior feature data is collected, which can be divided into two categories in terms of type. The first category is sequence features, and the second category is behavior statistics features. The sequence features are divided into two parts, namely, a user operation access sequence {opt1, opt2..} representing a page sequence accessed by the user within a period of time, and a page access time sequence {tdf1, tdf2..} representing the user accessing different pages. A crawler protection model based on LSTM is designed, which includes an LSTM layer, a neural network layer, and a full connection layer. The LSTM layer includes input gate, forget gate, and output gate control units, is suitable for processing long sequence information, and can selectively retain important information existing in the long sequence. The crawler protection model structure is shown in Figure 4 , Figure 5 A data processing flow of the intelligent crawler protection model based on the LSTM algorithm given in the embodiment is provided, which mainly describes the processing process of data in the LSTM model. F t represents the forget gate, I t represents the input gate, O t represents the output gate, the functions of the three gates mainly control the input and output and selection of data, C t represents the memory of the model, is used to record the input information, and outputs the processed information H t after processing by an activation function σ. The neural network layer is used to process user behavior statistics features, and is used to mine potential behaviors of malicious crawlers, such as page access time intervals and historical payment ratios. The full connection layer can retain important information of the network, is used to fuse the output results of the LSTM layer and the neural network layer, and outputs the final crawler score after multiple convolutional layers.

[0078] The present disclosure also provides a crawler protection method based on the above crawler protection system, which includes:

[0079] The crawler protection intelligent agent is used to: transmit access request information to the crawler protection model; receive analysis information of the machine learning platform on the access request information; and execute a crawler protection strategy corresponding to the analysis information.

[0080] In summary, the technical solutions and technical concepts of the present disclosure are as follows:

[0081] 1. Overall, the present solution architecture is designed to design an anti-crawler strategy, which is divided into two parts. The first part is to identify malicious users by combining expert strategies and machine learning models, and the second part is to set risk response strategies for users with different risk levels.

[0082] The first part of the embodiment is to set different layers for data burying and identification of malicious user requests. First, a channel layer based on an SDK method is set, which is used to collect user request data and perform basic filtering on the user request data, and then interact with the intelligent prevention and control system to obtain expert strategies, and perform operations such as shunting, human-computer verification identification, and access interception according to the expert strategies. The channel layer also outputs logs according to the requirements of machine learning model training, such as the IP of the access user, the cookie information of the user, the request time of the user page access, the request size, etc. When the expert strategy cannot complete the scoring of the user, the actual analysis of the user request information data is performed by calling a machine learning inference service to perform risk scoring of the request. The process is as shown in FIG. 8. Figure 6

[0083] The second part is to set different prevention and control strategies for access users with different malicious degrees, and to return the malicious value (request score) of the current IP / user / device fingerprint by calling the prevention and control client / server / machine learning platform. The malicious value is in percentage, and is set to five grades. Due to the accuracy of the crawler in identifying the malicious value, as the malicious value increases, a more and more accurate parabolic phenomenon will occur. Therefore, the interception rule adopts the strategy of the tonal relationship in photography, which is divided into five large area strategy ranges of white, highlight, neutral gray, shadow, and black. The requests of new IP, new user, and new device adopt relatively loose business prevention and control strategies such as high light, and the original platform flight query strategy is still used for flight query, and the forced prevention and control strategy is not used for seat occupation, or relatively loose strategies such as shortening the payment time are used.

[0084] 2. The scheme architecture is used to deal with batch data generation. Two parts of tasks are designed, one is the management of the task, which divides and checks the task, and the system determines the optimal amount of data to be processed in a batch in the design, which needs to be considered comprehensively according to the size of a single data; the other is the specific execution part of the task, which processes the storage of data to the database.

[0085] The first part of the real-time scheme is to ensure that the task can be divided into a relatively small granularity for further smooth processing, and to ensure that all data is completely processed; first, the data to be generated in this batch is divided to obtain the total batch quantity to be divided. Then, a task data record is generated to record the following contents: total quantity, scale, task completion status, serial number of each batch, and execution status. The data is recorded in the database for persistence. At the same time, a timing task is set, which can be triggered according to the frequency of business time requirements, and after completion of the recording, the execution part of the task is called.

[0086] ​The second part of the specific embodiment is to ensure that each batch of split data is completely stored in the database, and at the same time, it is ensured that the batch task has been confirmed to be completed. A certain amount of data is extracted from the data id, assembled into a record that needs to be stored in the database to generate sql, and the task data record is modified to change the current batch represented to executed, provided that the initial state of the batch is not executed. Generate sql, submit the database for execution in one sql, and process it in one transaction. In this way, it can be ensured that the data and task data are updated at the same time or fail, and the amount of processed data and the task progress are consistent.

[0087] 3. This scheme architecture is designed in three parts when training through machine learning model. One is the reading and storage of data, centralized management of multi-type data from internal and external sources, to ensure data quality, security and compliance. Two is the training and optimization of the model, providing one-stop service, simplifying the workflow of engineers, accelerating the model building, testing and iteration process. Three is the unified service of the model, which enables the trained model to be quickly and stably deployed to the production environment, seamlessly integrated with the existing business system, and empowered products and services.

[0088] The first part of the specific embodiment is to design a DataServer module in the machine learning platform, which is responsible for providing data filtering, data collection, data integration, data cleaning and other data services. According to the filtering conditions, data and text data are collected from the database system for data integration, and the preprocessed data is cleaned to ensure that the data has the correct format, such as converting string to date, normalizing or standardizing numerical data, etc. According to the query data conditions, the data interface provided by the database system is requested using Http / Https protocol to obtain data. Data server completes the integration and cleaning of data, uploads to feature center in batches, and provides data support for algorithm model. Streaming data uses middleware (kafka MQ, etc.) for temporary storage.

[0089] The second part of the embodiment is to design an algorithm training module on the machine learning platform. A developer fills in an application resource list (GPU, CPU, memory, etc.) by accessing an Admin Server service through a gateway (Https / Http). The Admin Server uses a GRPC protocol to request a dispatch center to automatically match the applied resources to appropriate resource servers. The Admin Server creates a workspace (Docker remote call) on the resource servers. The developer accesses a jupyter-lab of the workspace through a gateway (Https / Http, Websocket protocol reverse proxy) to do algorithm experiments and model verification. An Admin Server visual UI shows (GRPC protocol request) a monitor service to monitor all workspaces and model inference services. The monitor service performs relevant operations (whether to delete the workspace) by querying the state of the workspace in a DB (whether the application use time is expired) and monitoring (using an http protocol to access a model health check interface and a Docker remote call to check a container running state) the corresponding container.

[0090] The third part of the embodiment is to design a model deployment and service module on the machine learning platform. A container running environment is configured on a specified resource server. An image (environment dependencies including algorithms and frameworks such as Pytorch and cuda), port mapping, resource limit, and other parameters of the container are specified. A corresponding container is started on a corresponding server through a dispatch center. A container startup command uses an Mlflow model deployment instruction Mlflow models server to start a model to provide external services. According to the size of the model file, two methods can be used. The model file can be stored in a minio cluster and downloaded through an s3 protocol when the container is started. For a large model, the model file can be uploaded to a target server first. A container and a host volume (data volume) are mounted and mapped to start a large model service. The model is added to a monitoring service monitoring list. The monitoring service periodically checks the model. The monitoring service calls a health check interface provided by the model to ensure that the model inference can normally provide external services. The monitoring service also checks the container running state to ensure the stable operation of the container layer.

[0091] In addition, as shown in Figure 7 The electronic device 1000 includes a memory 1002 and a processor 1001. The memory 1002 stores a computer program or instructions. When the computer program or instructions are executed by the processor 1001, at least the crawler protection method described above is implemented. As shown in Figure 8As shown, the present disclosure provides a computer readable storage medium 1100, in which a computer program or instructions are stored, and the computer program or instructions are executed by a processor to at least implement the above-described crawler protection method.

[0092] In addition, the present disclosure also provides a computer program product stored in a computer readable storage medium, and the computer program product is executed by a processor to at least implement the above-described crawler protection method.

[0093] Obviously, various modifications and variations of the present disclosure can be made by those skilled in the art without departing from the spirit and scope of the present disclosure. Thus, it is intended that the present disclosure also cover the modifications and variations of this disclosure to fall within the scope of the claims of the present disclosure and their equivalents.

Claims

1. A reptile protection system, characterized in that: include: Reptile protection agent and LSTM-based reptile protection model Reptile protection model; Among them, The crawler protection agent is used to: Transmitting access request information to the crawler protection model; receiving parsed information of the access request information by the crawler protection model, and determining a malicious value of the access request information; Executing a crawler protection strategy corresponding to the malicious value; wherein the malicious value is divided into several levels, each level corresponding to a crawler protection strategy; The training process of the crawler protection model includes: Collect behavioral characteristic data of business-end users; A crawler protection model is trained based on the behavioral feature data; wherein the crawler protection model is composed of an LSTM layer, a neural network layer, and a fully connected layer connected in sequence; the LSTM layer includes: an input gate, a forget gate, and an output gate gating unit; Build a crawler protection model based on the trained crawler protection model; The behavioral characteristic data includes: Sequence characteristics and behavioral statistical characteristics; among them, The sequence features include: access operation sequence and access page sequence; The LSTM layer of the crawler protection model contains two LSTM networks, which are used to process the access operation sequence and the access page sequence respectively; and The neural network layer is used to process the behavioral statistical features.

2. The reptile protection system according to claim 1, wherein: The crawler protection agent includes: Reptile protection business end and reptile protection service end; Among them, The crawler protection service end is used to perform service end crawler protection according to the service end expert protection strategy; wherein the service end crawler protection includes: Whitelisting, requesting release, increasing difficulty, periodic bans, and permanent bans; The crawler protection server is used to perform server-side crawler protection according to the server-side expert protection strategy; wherein the server-side crawler protection includes: Based on the server-side expert protection strategy, the access request is scored.

3. The reptile protection system according to claim 2, wherein: The server-side crawler protection also includes: Sending access request information corresponding to the access request to the crawler protection model; and Receive the access request score value returned by the crawler protection model.

4. The reptile protection system according to claim 2, wherein: The server-side crawler protection also includes: The access request is processed accordingly according to the score value of the access request.

5. The reptile protection system according to claim 1, wherein: The crawler protection agent is also used to perform crawler identification based on big data processing technology, and the execution method includes: Regularly capture log data corresponding to access requests; Performing standardization and data cleaning on the log data; Perform aggregate analysis on the processed data to determine whether the current access request comes from a crawler.

6. A reptile protection method based on the system according to any one of claims 1 to 5, characterized in that: include: Using the crawler protection agent: Transmitting access request information to the crawler protection model; Receiving parsing information of the access request information by the crawler protection model; Execute the crawler protection strategy corresponding to the parsed information.

7. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program or instructions, which, when executed by a processor, are used to at least implement the method of claim 6.

Images