Crawler protection system and method and computer readable storage medium

Abstract

The invention relates to the technical field of network security, in particular to a crawler protection system and method and a computer readable storage medium, and the system comprises a crawler protection agent and an LSTM-based crawler protection model; wherein the crawler protection intelligent agent is used for transmitting access request information to the crawler protection model; receiving analysis information of the crawler protection model on the access request information, and determining a malicious value of the access request information; and executing a crawler protection strategy corresponding to the malicious value. According to the method, real-time access sequence features, time features and behavior statistical features are fully utilized, the functions of data storage and management, data preprocessing and feature extraction, real-time data updating and the like are provided for intelligent crawler protection model training through efficient design of data storage, an efficient algorithm is designed based on an efficient data management mode, and the training efficiency is improved. Abnormity recognition is carried out on the user request in real time, and prevention and control measures are taken in time.

Description

Technical Field

[0001] The present disclosure relates to the field of network security technology, and in particular to a crawler protection system, method and computer-readable storage medium. Background Art

[0002] With the advent of the digital and networked era, information security issues have gradually come into people's view. With the rapid development of crawler technology, web crawlers, as a data collection tool, capture a large amount of website background data, often causing the leakage of website information and personal information. In particular, the activities of malicious crawlers have posed a serious threat to the information security of the civil aviation field. The information security of the civil aviation field is facing unprecedented challenges.

[0003] Traditional crawler protection technologies are often rule-based, such as IP blocking, User-Agent verification, and Cookie verification. The main means are to block suspicious IPs using blacklists and whitelists, determine disguised request information through request header information, and verify whether the Cookie value is abnormal. Although these methods are simple and direct to implement, they can block specific IP crawler requests or poorly disguised crawler requests. However, these strategies have the risk of misjudging normal users and may affect the experience of normal users. In severe cases, it will lead to user loss for airlines and loss of revenue. Moreover, with the continuous iterative development of web crawler technology, the traditional rule-based crawler protection mode can no longer meet the requirements of ensuring information security, and traditional strategies also need to be continuously updated and iterated. Summary of the invention

[0004] The present disclosure aims to solve at least one of the technical problems in the above-mentioned technologies to a certain extent, and proposes a reptile protection system, comprising:

[0005] Reptile protection agent and LSTM-based reptile protection model;

[0006] The crawler protection agent is used to:

[0007] Transmitting access request information to the crawler protection model;

[0008] Receiving parsing information of the access request information by the crawler protection model, and determining a malicious value of the access request information;

[0009] Execute the crawler protection strategy corresponding to the malicious value.

[0010] Furthermore, the crawler protection agent includes:

[0011] Crawler protection business end and crawler protection service end; among them,

[0012] The crawler protection service end is used to perform service end crawler protection according to the service end expert protection strategy; wherein the service end crawler protection includes:

[0013] Whitelisting, requesting release, increasing difficulty, periodic ban, and permanent ban;

[0014] The crawler protection server is used to perform server-side crawler protection according to the server-side expert protection strategy; wherein the server-side crawler protection includes:

[0015] Based on the server-side expert protection strategy, the access request is scored.

[0016] Furthermore, the server-side crawler protection further includes:

[0017] Sending access request information corresponding to the access request to the crawler protection model; and

[0018] Receive the access request score value returned by the crawler protection model.

[0019] Furthermore, the server-side crawler protection further includes:

[0020] The access request is processed accordingly according to the score value of the access request.

[0021] Furthermore, the crawler protection agent is also used to perform crawler identification based on big data processing technology, and the execution method includes:

[0022] Regularly capture log data corresponding to access requests;

[0023] Performing standardization and data cleaning on the log data;

[0024] Perform aggregate analysis on the processed data to determine whether the current access request comes from a crawler.

[0025] Furthermore, the training process of the crawler protection model includes:

[0026] Collect behavioral characteristic data of business end users;

[0027] The crawler protection model is trained according to the behavior feature data; wherein the crawler protection model is composed of an LSTM layer, a neural network layer and a fully connected layer connected in sequence; the LSTM layer includes: an input gate, a forget gate and an output gate gating unit;

[0028] Build a crawler protection model based on the trained crawler protection model.

[0029] Furthermore, the behavior characteristic data includes:

[0030] Sequence characteristics and behavioral statistical characteristics; among them,

[0031] The sequence features include: an access operation sequence and an access page sequence.

[0032] Furthermore, the LSTM layer of the crawler protection model contains two LSTM networks, which are used to process the access operation sequence and the access page sequence respectively; and

[0033] The neural network layer is used to process the behavioral statistical features.

[0034] The present disclosure also provides a reptile protection method based on the above reptile protection system, comprising:

[0035] Using the crawler protection agent:

[0036] Transmitting access request information to the crawler protection model;

[0037] Receiving parsing information of the access request information by the crawler protection model;

[0038] Execute the crawler protection strategy corresponding to the parsed information.

[0039] The present disclosure also proposes a computer-readable storage medium, in which a computer program or instruction is stored. When the computer program or instruction is executed by a processor, it is at least used to implement the above method.

[0040] Compared with the prior art, the beneficial effects of the present invention are:

[0041] The present invention makes full use of real-time access sequence features, time features, and behavioral statistical features, and through efficient data storage design, provides data storage and data management, data preprocessing and feature extraction, real-time data update and other functions for intelligent crawler protection model training, and designs efficient algorithms based on efficient data management models to identify anomalies of user requests in real time and take timely prevention and control measures.

[0042] Other features and advantages of the present disclosure will be described in the subsequent description, and partly become apparent from the description, or be understood by practicing the present disclosure. The purpose and other advantages of the present disclosure can be realized and obtained by the structures particularly pointed out in the written description and the drawings. The technical solution of the present disclosure is further described below through drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The accompanying drawings are used to provide a further understanding of the present disclosure and constitute a part of the specification. Together with the embodiments of the present disclosure, they are used to explain the present disclosure and do not constitute a limitation of the present disclosure. In the accompanying drawings:

[0044] Figure 1 A schematic diagram of a reptile protection system provided for this application;

[0045] Figure 2 A schematic diagram of a reptile protection system provided in an embodiment;

[0046] Figure 3 The training flow chart of the crawler protection model given in the embodiment;

[0047] Figure 4 A schematic diagram of the structure of a reptile protection model provided in an embodiment;

[0048] Figure 5 A schematic diagram of the crawler protection model data processing flow provided in the embodiment;

[0049] Figure 6 The following is a flowchart of the operation of the crawler protection model provided in the embodiment;

[0050] Figure 7 A schematic diagram of an electronic device provided for an embodiment;

[0051] Figure 8 A schematic diagram of a computer-readable storage medium is provided for an embodiment. DETAILED DESCRIPTION

[0052] The present disclosure is described below in conjunction with the accompanying drawings. The preferred embodiments described herein are only used to illustrate and explain the present disclosure and are not used to limit the present disclosure.

[0053] like Figure 1 As shown, the present disclosure provides a reptile protection system, comprising:

[0054] Reptile protection agent and LSTM-based reptile protection model;

[0055] The crawler protection agent is used to:

[0056] Transmitting access request information to the crawler protection model;

[0057] Receiving parsing information of the access request information by the crawler protection model, and determining a malicious value of the access request information;

[0058] Execute the crawler protection strategy corresponding to the malicious value.

[0059] Furthermore, the crawler protection agent includes:

[0060] Crawler protection business end and crawler protection service end; among them,

[0061] The crawler protection service end is used to perform crawler protection on the service end according to the service end expert protection strategy; the crawler protection on the service end includes: whitelist labeling, request release, increase difficulty, periodic ban and permanent ban;

[0062] The crawler protection server is used to perform server-side crawler protection according to the server-side expert protection strategy; the server-side crawler protection includes: scoring access requests based on the server-side expert protection strategy.

[0063] Furthermore, the server-side crawler protection also includes: sending access request information corresponding to the access request to the crawler protection model; and receiving the access request score value returned by the crawler protection model.

[0064] Furthermore, the server-side crawler protection also includes: processing the access request accordingly according to the score value of the access request.

[0065] According to some embodiments of the present disclosure, Figure 2 The intelligent crawler protection system consists of two parts: the intelligent crawler protection agent and the machine learning platform. The machine learning platform is a model training platform for developing crawler protection models, and its core lies in the crawler protection model based on LSTM. The intelligent crawler protection system business agent consists of the intelligent crawler protection business end and the intelligent crawler protection server end. The intelligent crawler protection business end performs five types of operations, including whitelisting, request release, difficulty increase, periodic ban, and permanent ban, based on the relatively basic expert protection strategy in the client. The intelligent crawler protection server end is composed of the server-side expert strategy scoring and the machine learning platform scoring, which jointly intercept malicious crawlers and malicious access requests. The machine learning platform stores user behavior feature data sets in the feature center for machine learning model training. The trained machine learning model provides services in a unified manner and gives the risk score of the request in real time based on the input of the intelligent crawler protection system.

[0066] Furthermore, the crawler protection agent is also used to identify crawlers based on big data processing technology, including:

[0067] Regularly capture log data corresponding to access requests;

[0068] Performing standardization and data cleaning on the log data;

[0069] Perform aggregate analysis on the processed data to determine whether the current access request comes from a crawler.

[0070] According to some embodiments of the present disclosure, the crawler identification technology based on big data is as follows: first, the log data stored in es is captured in real time through Sqoop to ensure the timeliness and integrity of the data, and then the collected logs are standardized and converted into a unified format and stored on the big data platform hdfs (distributed file system). Secondly, a table is built in the hive data warehouse according to the log table structure, stored in the original layer ods, and the data is mapped to the ods table. Through data cleaning, data conversion and other data processing, the log is desensitized, and the null values, outliers, duplicate logs, missing values, etc. in the data table are removed to improve data quality. The processed data is stored in the dwd layer as an intermediate table, and then the intermediate table is analyzed to extract the required fields, and the required data is aggregated at the same time, and the total number, average value, etc. are calculated. Finally, the calculated results are stored in the ads layer for labeling, and the current IP is judged as a crawler according to the obtained indicators.

[0071] Furthermore, the training process of the crawler protection model includes:

[0072] Collect behavioral characteristic data of business end users;

[0073] The crawler protection model is trained according to the behavior feature data; wherein the crawler protection model is composed of an LSTM layer, a neural network layer and a fully connected layer connected in sequence; the LSTM layer includes: an input gate, a forget gate and an output gate gating unit;

[0074] Build a crawler protection model based on the trained crawler protection model.

[0075] Furthermore, the behavior feature data includes: sequence features and behavior statistical features; wherein the sequence features include: access operation sequence and access page sequence.

[0076] Furthermore, the LSTM layer of the crawler protection model contains two LSTM networks, which are used to process the access operation sequence and the access page sequence respectively; the neural network layer is used to process the behavioral statistical characteristics.

[0077] According to some embodiments of the present disclosure, Figure 3This is the training flow chart of the intelligent crawler protection model of the machine learning platform. The machine learning platform is built based on the crawler protection algorithm. The construction process of the crawler protection algorithm is as follows: Collect user behavior feature data, which can be divided into two categories. The first category is sequence features, and the second category is behavioral statistical features. Sequence features are divided into two parts, namely, the user operation access sequence {opt1, opt2..} represents the sequence of pages visited by the user within a period of time, and the page access time series {tdf1, tdf2..} represents the user's visits to different pages. Design a crawler protection model based on LSTM, which includes three parts: LSTM layer, neural network layer, and fully connected layer. The LSTM layer contains input gate, forget gate, and output gate gating units, which are suitable for processing long sequence information and can selectively retain important information in long sequences. The structure of the crawler protection model is as follows Figure 4 As shown, Figure 5 The data processing flow of the intelligent crawler protection model based on the LSTM algorithm given in the embodiment mainly describes the data processing process inside the LSTM model, where F t represents the forget gate, I t represents the input gate, O t Represents the output gate. The functions of the three gates are mainly to control the input, output and selection of data. t Represents the memory of the model, which is used to record input information and output the processed information H after being processed by the activation function σ t The neural network layer is used to process the statistical characteristics of user behavior and to mine the potential behavior of malicious crawlers, such as the time interval between page visits and the historical payment ratio. The fully connected layer can retain important information of the network and is used to fuse the output results of the LSTM layer and the neural network layer. After multiple layers of convolution, the final predicted crawler score is output.

[0078] The present disclosure also provides a reptile protection method based on the above reptile protection system, comprising:

[0079] Utilize the crawler protection agent: transmit access request information to the crawler protection model; receive the analysis information of the access request information by the machine learning platform; and execute the crawler protection strategy corresponding to the analysis information.

[0080] In summary, the technical solutions and technical concepts of the present disclosure are as follows:

[0081] 1. Overall, the architecture of this solution is divided into two parts when designing the anti-air crawler strategy. The first part is to identify malicious users by combining expert strategies and machine learning models, and the second part is to set risk response strategies for users with different risk levels.

[0082] The specific implementation plan for the first part is to set up different layers to track data and identify malicious user requests. First, set up a channel layer based on the SDK method, track data in each application, and collect user request data. After basic filtering of the user request data, interact with the intelligent prevention and control system to obtain expert strategies, and perform diversion, human-machine verification and identification, access interception and other operations based on the expert strategies. The channel layer will also output logs according to the requirements of machine learning model training, such as the IP address of the accessing user, the user's cookie information, the request time of the user's page access, the request size, etc. If the expert strategy cannot complete the scoring of the user, the actual analysis of the user's request information data will be used to call the machine learning reasoning service to perform the risk scoring of the request. The process is as follows: Figure 6 shown.

[0083] The second part is to set different prevention and control strategies for visiting users with different degrees of malice. By calling the prevention and control client / server / machine learning platform, the malicious value (request score) of the current IP / user / device fingerprint will be returned. The malicious value is in percentage and is set to 5 levels. Due to the accuracy of crawlers in identifying malicious values, as the malicious value increases, a parabolic phenomenon will become more and more accurate. Therefore, the interception rules adopt the strategy of tonal relationship in photography, which is divided into five large regional strategy ranges: white, highlight, neutral gray, shadow, and black. For requests from new IPs, new users, and new devices, relatively loose business prevention and control strategies such as highlights are adopted. When inquiring about flights, the original strategy of flight inquiries on the original platform is still adopted. To prevent seat occupation, mandatory prevention and control strategies are not used, or relatively loose strategies such as shortening payment time are used.

[0084] 2. This solution architecture is designed to deal with batch data generation. Two parts of tasks are designed: one is the management of tasks, task splitting and inspection. The system is designed to determine how much data is better to process in a batch, and the size of a single data needs to be considered comprehensively; the other is the specific execution part of the task, which processes the storage of data in the database.

[0085] The specific real-time solution for the first part of the task is to ensure that the task can be split into relatively fine granularity for further smooth processing, and at the same time to ensure that the data is finally completely processed; first split the data to be generated in this batch to obtain the total batch size to be split. Then generate the task data record, recording the following contents: total quantity, scale, task completion status, serial number and execution status of each batch. Record this data in the database for persistence. At the same time, set a scheduled task. This time can be based on the urgency of the business time requirements to control the trigger frequency. After completing the entry, call the execution part of the task.

[0086] The specific implementation plan for the second part is to ensure that each batch of split data can be completely stored in the database, and at the same time ensure that the batch task has been confirmed to be completed. Extract the measured data from the data id, assemble it into records that need to be stored in the database to generate SQL, and modify the task data record at the same time, change the current batch represented to executed, the condition is that the initial state of this batch is not executed, generate SQL, submit it to the database for execution together, and process it in one transaction. This ensures that these data and task data are updated at the same time or fail, and ensures that the amount of data processed and the task progress are consistent.

[0087] 3. This solution architecture is designed with three parts when training through machine learning models. The first is the reading and storage of data, which centrally manages multiple types of data from internal and external sources to ensure data quality, security and compliance. The second is model training and tuning, providing one-stop services, simplifying engineers' workflows, and accelerating model building, testing and iteration processes. The third is the unified service provision of the model, so that the trained model can be quickly and stably deployed to the production environment, seamlessly integrated with the existing business system, and empowering products and services.

[0088] The first part of the specific implementation plan designs the DataServer module on the machine learning platform, which is responsible for providing data services such as data screening, data collection, data integration, and data cleaning. According to the screening conditions, data is collected from the database system and the text data is integrated, and the pre-processed data is cleaned to ensure that the data has the correct format, such as converting strings into dates, normalizing or standardizing numerical data, etc. According to the query data conditions, the Http / Https protocol is used to request the data interface provided by the database system to obtain data. The Data server completes the integration and cleaning of the data, and uploads it to the feature center in batches to provide data support for the algorithm model. Streaming data is temporarily stored using middleware (kafka MQ, etc.).

[0089] The specific implementation plan for the second part is to design an algorithm training module on the machine learning platform. Developers access the Admin Server service through the gateway (Https / Http) to fill in the application resource list (GPU, CPU, memory, etc.). The Admin Server uses the GRPC protocol to request the scheduling center to automatically match the resources requested to the appropriate resource server and create a workspace on the resource server (Docker remote call). Developers access the jupyter-lab in the workspace through the gateway (Https / Http, Websocket protocol reverse proxy) to perform algorithm experiments and model verification. The Admin Server visual UI displays (GRPC protocol request) the monitoring status of the monitor service for all workspaces and model inference services. The monitor service queries the status of the workspace in the DB (whether the application usage time has expired), monitors (using the http protocol to access the model health check interface, and Docker remote calls to check the container running status) the corresponding container to perform related operations (whether to delete the workspace).

[0090] The specific implementation plan of the third part is to design the model deployment and service modules on the machine learning platform. Configure the container operating environment on the specified resource server, specify the container image (environmental dependencies include algorithms and frameworks such as Pytorch and cuda), port mapping, resource restrictions and other parameters. Start the corresponding container on the corresponding server through the scheduling center. The container startup command uses the Mlflow deployment model instruction Mlflow models server to start the model and start external services. According to the size of the model file, two methods can be used. The model stored in the minio cluster can be downloaded through the s3 protocol when the container is started. For large models, the model file can be uploaded to the target server first, and the large model service can be started by mounting and mapping the container and the host volume (data volume). Add it to the monitoring list of the monitoring service, and the monitoring service will check it periodically. The model provides a health check interface for the monitoring service to call to ensure that the model reasoning can provide external services normally. The monitoring service will also check the running status of the container to ensure the stable operation of the container layer.

[0091] In addition, if Figure 7 As shown, the present disclosure provides an electronic device 1000, which includes a memory 1002 and a processor 1001. The memory 1002 stores a computer program or instruction. When the computer program or instruction is executed by the processor 1001, it is at least used to implement the above-mentioned crawler protection method. Figure 8As shown, the present disclosure provides a computer-readable storage medium 1100, in which a computer program or instruction is stored. When the computer program or instruction is executed by a processor, it is at least used to implement the above-mentioned crawler protection method.

[0092] In addition, the present disclosure also provides a computer program product, which is stored in a computer-readable storage medium. When the computer program product is executed by a processor, it is at least used to implement the above-mentioned crawler protection method.

[0093] It is obvious that a person skilled in the art can make various changes and modifications to the present disclosure without departing from the spirit and scope of the present disclosure. Thus, if these modifications and variations of the present disclosure fall within the scope of the claims of the present disclosure and their equivalents, the present disclosure is also intended to include these modifications and variations.

Claims

1. A reptile protection system, characterized in that: include: Reptile protection agent and LSTM-based reptile protection model Reptile protection model; Among them, The crawler protection agent is used to: Transmitting access request information to the crawler protection model; Receiving parsing information of the access request information by the crawler protection model, and determining a malicious value of the access request information; Execute the crawler protection strategy corresponding to the malicious value.

2. The reptile protection system according to claim 1, characterized in that: The crawler protection agent comprises: Crawler protection business end and crawler protection service end; among them, The crawler protection service end is used to perform service end crawler protection according to the service end expert protection strategy; wherein the service end crawler protection includes: Whitelisting, requesting release, increasing difficulty, periodic ban, and permanent ban; The crawler protection server is used to perform server-side crawler protection according to the server-side expert protection strategy; wherein the server-side crawler protection includes: Based on the server-side expert protection strategy, the access request is scored.

3. The reptile protection system according to claim 2, characterized in that: The server-side crawler protection also includes: Sending access request information corresponding to the access request to the crawler protection model; and Receive the access request score value returned by the crawler protection model.

4. The reptile protection system according to claim 2, characterized in that: The server-side crawler protection also includes: The access request is processed accordingly according to the score value of the access request.

5. The reptile protection system according to claim 1, characterized in that: The crawler protection agent is also used to perform crawler identification based on big data processing technology, and the execution method includes: Regularly capture log data corresponding to access requests; Performing standardization and data cleaning on the log data; Perform aggregate analysis on the processed data to determine whether the current access request comes from a crawler.

6. The reptile protection system according to claim 5, characterized in that: The training process of the crawler protection model includes: Collect behavioral characteristic data of business end users; The crawler protection model is trained according to the behavior feature data; wherein the crawler protection model is composed of an LSTM layer, a neural network layer and a fully connected layer connected in sequence; the LSTM layer includes: an input gate, a forget gate and an output gate gating unit; Build a crawler protection model based on the trained crawler protection model.

7. The reptile protection system according to claim 6, characterized in that: The behavioral characteristic data includes: Sequence characteristics and behavioral statistical characteristics; among them, The sequence features include: an access operation sequence and an access page sequence.

8. The reptile protection system according to claim 7, characterized in that: The LSTM layer of the crawler protection model contains two LSTM networks, which are used to process the access operation sequence and the access page sequence respectively; and The neural network layer is used to process the behavioral statistical features.

9. A method for protecting against reptiles based on the system according to any one of claims 1 to 8, characterized in that: include: Using the crawler protection agent: Transmitting access request information to the crawler protection model; Receiving parsing information of the access request information by the crawler protection model; Execute the crawler protection strategy corresponding to the parsed information.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program or instruction, and when the computer program or instruction is executed by the processor, it is used to implement at least the method of claim 9.

Images