A machine learning-based crawler behavior detection method for map tile services

Abstract

The application provides a crawler behavior detection method for a map slice service based on machine learning. The method starts from the mechanism of the crawler for the map slice service, deeply mines the content information of the user request traffic data through a feature engineering method, sufficiently describes the crawler behavior, simultaneously utilizes a LightGBM framework to establish a machine learning model, efficiently fits and analyzes the features obtained by mining the traffic data, and finally realizes accurate identification of the crawler behavior for the map slice service.

Description

Technical Field

[0001] This invention relates to the field of map tile service technology, and in particular to a machine learning-based method for detecting crawler behavior in map tile services. Background Technology

[0002] Map data is typically stored in large Geographic Information System (GIS) databases, including map layers, roads, terrain, place names, etc. Map tile services are services that divide map data into small tiles (tiles) and provide them to map applications in a fast-loading manner. These tiles are typically stored in image formats (such as PNG or JPEG) or vector formats (such as GeoJSON) and can be dynamically loaded based on the map's zoom level and location. The primary goal of map tile services is to provide high-performance and efficient map data transmission for displaying interactive maps in web and mobile applications.

[0003] A web crawler is an automated program or script used to browse, retrieve, and extract information from the internet. By following links and downloading content from web pages, web crawlers can obtain vast amounts of data from the internet, which can be used for various purposes, including search engine indexing, data mining, price comparison, news aggregation, and social media analysis. However, when web crawlers are abused or misused, they can pose some potential harms to map services, including:

[0004] 1. Excessive Data Requests: Web crawlers can send a large number of requests at a very high speed, which can increase the server load of map services. Without proper limits and protection mechanisms, this can lead to a decline in server performance and affect the experience of normal users.

[0005] 2. Misuse of map data: Web crawlers can obtain map data from map services, including map layers, satellite imagery, and geographic information. This data may be used for unauthorized commercial purposes, such as creating competing map services without the data provider's permission.

[0006] 3. Data corruption and misuse: Some malicious web crawlers may send false or malicious requests in an attempt to damage map services or cause data leaks. Such behavior may threaten the security and integrity of map data.

[0007] Therefore, it is necessary to build a crawler detection model for map tile services to detect and identify malicious crawler users.

[0008] With the widespread application of map tile services across various industries, crawler behavior detection has become a crucial link in ensuring the normal operation of services and data security. Map tile service providers are increasingly emphasizing the supervision of crawler behavior. Most existing methods for detecting crawler behavior for map tile services involve analyzing traffic data, typically including analyzing access patterns, detecting abnormal request frequencies, verifying user agents, monitoring IP addresses, and limiting application access rates. Ultimately, manually defined rules and policies are used to detect abnormal crawler behavior.

[0009] However, existing rule-based crawler behavior detection methods have two problems: First, the detection methods based on rule strategies are relatively fixed, and the timeliness of the detection mode is poor. If the crawler strategy changes, the detection rule strategy will become invalid, and it will be unable to effectively detect the new crawler strategy. Second, rule-based crawler behavior detection methods only consider the feature structure of the traffic data surface and do not conduct in-depth mining of the accessed content information. As a result, the modeling of crawler behavior is not sufficient, and ultimately the detection method has a low recall rate for crawler behavior and cannot effectively identify crawler behavior.

[0010] For example, existing technology discloses a method and system for identifying map tile data crawlers, specifically application number 202111132457.4, and the specific flowchart is as follows. Figure 1 As shown. In this patent application, S2-S5 preprocess the traffic data stream and add timestamps, group the traffic data using IP addresses as key-value pairs, and then set a fixed time window to divide the data stream. S6 designs the following rules to detect crawler behavior:

[0011] Count the number of data entries containing the same IP address within each time window, denoted as x; count the number of data entries within each time window using "(IP, map tile matrix name, map level)" as the key, denoted as y; count the number of data entries within each time window using "(IP, map tile matrix name, map level, map tile row number, map tile column number)" as the key, denoted as z. In the following two logics, if any action of the current IP address satisfies either logic, the current IP address can be identified as a crawler IP.

[0012] Logic 1: Within the time window, the condition a1×x+a2×y≥θ1 is satisfied, where the parameters a1, a2, and θ1 must satisfy the conditions a1+a2=1, a1≥0, a2≥0, and θ1>0;

[0013] Logic 2: Within the time window, b1×x+b2×z≥θ2 is satisfied, where the parameters b1, b2, and θ2 must satisfy the conditions b1+b2=1, b1≥0, b2≥0, and θ2>0;

[0014] Compared with the present invention, this method has the following three problems:

[0015] 1. Before applying this method, multiple hyperparameters need to be manually assigned values, such as a1, a2, b1, b2, θ1, θ2 and time window size. The manually assigned parameters require a lot of experience and prior knowledge, and are difficult to apply directly.

[0016] 2. The design idea of ​​this method is to measure the amount of traffic requested by a user per second. If a user requests a lot of traffic per second, he is judged as an abnormal user. However, actual research has found that some crawler users will deliberately reduce the data crawling rate in order to avoid the blocking policy of the map service platform. Therefore, this method is difficult to identify these crawler users.

[0017] 3. This method only considers the number of requests per second from users for measurement, without conducting deeper analysis of user behavior. As a result, the modeling of crawler behavior is insufficient, ultimately leading to a low recall rate for crawler behavior and an inability to effectively identify crawler behavior. Summary of the Invention

[0018] The purpose of this invention is to overcome the shortcomings of existing technologies and propose a machine learning-based method for detecting crawler behavior in map tile services. This method starts from the mechanism of crawlers targeting map tile services, and uses feature engineering to deeply mine the content information of user request traffic data to fully describe crawler behavior. Simultaneously, it utilizes the LightGBM framework to build a machine learning model, performing efficient fitting analysis on the features obtained from traffic data mining, ultimately achieving accurate identification of crawler behavior in map tile services.

[0019] This invention is achieved through the following technical solution: This invention proposes a machine learning-based method for detecting crawler behavior in map tile services, the method comprising the following steps:

[0020] Step 1: Obtain traffic data: This step includes traffic logs obtained through a distributed message queue, file system, or database; each traffic data entry contains the following fields: user ID, request time timestamp, request source IP information, browser information, map tile matrix name, map zoom level, map tile row number, and map tile column number;

[0021] Step 2: User browsing session segmentation: The traffic data obtained in Step 1 is segmented into multiple data streams. Each data stream represents the traffic data generated by a user's operation behavior over a continuous period of time.

[0022] Step 3: Reproduction of user map browsing trajectory behavior: The data stream after session segmentation is parsed and calculated to obtain the user's map browsing trajectory during the entire session, which is then visualized in the form of a heat map;

[0023] Step 4: Construction of abnormal behavior training samples: If the visualization of a user's browsing trajectory for a certain session is neatly and densely arranged, forming a large and very regular rectangle, then that area represents abnormal data crawling behavior; otherwise, it represents a normal user. By manually selecting a portion of users' session browsing trajectories and manually annotating them, an abnormal behavior training sample dataset is obtained.

[0024] Step 5: User behavior feature extraction: The features include global traffic data features, traffic speed statistics features, operation behavior frequency distribution, traffic request time statistics features, user browsing behavior time statistics features, regional features of user browsing trajectory, traffic data information content measurement, and traffic data scaling interval level distribution, totaling 8 categories of features.

[0025] Step 6: Training the abnormal behavior detection model;

[0026] Step 7: Identify and archive abnormal users.

[0027] Furthermore, step 2 specifically includes:

[0028] Step 2.1: Group users: Group the traffic data by user ID or user access IP field in the traffic data. Each group of traffic data represents the traffic data generated by all operations of a user.

[0029] Step 2.2: Sort by request time: Sort each group of data in order of seconds according to the request timestamp of the traffic data, and aggregate the traffic data within each second to form a data stream in the form of key-value pairs with "user id, run timestamp" as the key and the set of traffic data per second as the value;

[0030] Step 2.3: Design threshold segmentation browsing sessions: Perform first-order time difference analysis on each data stream, calculate the time interval between each data stream segment and the previous data stream segment, and set the session maintenance interval T. gap If the time interval between two adjacent data streams is greater than the session maintenance interval, the data is divided using the interval between the two data streams as the boundary. Ultimately, each group of data streams forms a set of multiple data segments. Each segment is a key-value pair data stream with "user id, session id" as the key and each session traffic data set as the value.

[0031] Furthermore, step 3 specifically includes:

[0032] Step 3.1: parse the traffic data in seconds according to the request time order. Parse each data stream obtained in Step 2 and retain only the map zoom level, map tile row number and map tile column number fields.

[0033] Step 3.2: Determine the start time of the user operation behavior state. Define the user operation behavior as zoom in, zoom out, or pan. Then, sequentially traverse each data stream, comparing the map zoom level of each second-level time window data stream with the map zoom level of the next second-level time window data stream. If the two time window levels are the same, the user operation in the next time window is a pan operation; if the zoom level of the current time window data stream is greater than the zoom level of the next time window, the user operation in the next time window is a zoom out; if the zoom level of the current time window is less than the zoom level of the next second, the user operation in the next time window is a zoom in; if the user operation behavior changes between two adjacent time windows, or if the time interval between two adjacent time windows is greater than 2 seconds, the termination of one operation behavior and the start of another operation behavior can be determined. Mark an operation behavior with "user id, session id, operation behavior type, source map zoom level, target map zoom level, operation behavior start time, operation behavior end time".

[0034] Step 3.3: Geographical region discovery of user operation behavior. For the operation behavior identified in Step 3.2, obtain the data stream from the start time to the end time of the operation behavior. Only retain all data streams under the target map zoom level. The data format is "[(to_z,x_1,y_1),(to_z,x_2,y_2),…]", where to_z is the target map zoom level, x is the map tile column number, and y is the map tile row number. Integrate the data streams to form a two-dimensional array Matrix, and construct the corresponding coordinate mapping function f:(x,y)→(m,n). If the data stream contains data (x,y), then the value of Matrix[f(x,y)] is 1; otherwise, it is 0. After assigning values ​​to the two-dimensional array Matrix, set the size of the sliding rectangular window (size). x size y The points with a value of 1 in the two-dimensional array are slid and translated to obtain a browsing trajectory of an operation. Each browsing trajectory is marked with "operation, source target map zoom range, target map zoom range, slice coordinates of the lower left corner of the translation matrix, slice coordinates of the upper right corner of the translation matrix, and slice coordinates of the middle of the translation matrix".

[0035] Step 3.4: Integrate user operation behavior paths. Based on steps 3.2 and 3.3, determine all operation behaviors and browsing trajectories in a session, and then store each browsing trajectory in an ordered queue.

[0036] Step 3.5: Coordinate transformation of user operation path. The slice coordinates saved in steps 3.3 and 3.4 are in Mercator coordinate system. The slice coordinates need to be transformed into latitude and longitude coordinate system.

[0037] Step 3.6: Visualize user map browsing trajectory. Design the corresponding front-end interface to display the coordinates of the intermediate slices of the translation matrix of the user operation behavior path obtained in the previous steps in the form of a heat map.

[0038] Further, the global characteristics of the traffic data are described as follows: (1) Slice session duration, which refers to the timestamp of the last data stream of a session minus the timestamp of the first data stream; (2) Number of user browsing times, which refers to the number of times the time difference between two adjacent data streams in a session is greater than 2 seconds; (3) Total number of requested traffic data, which refers to the total number of traffic data requests from the user to the map slice server in a session; (4) Number of actual operation behaviors, which refers to the total number of times the user performs zoom-in, zoom-out, and panning operations in a session.

[0039] The traffic speed statistical characteristics are described as follows: (1) the total time spent by users requesting traffic data from the map tile server; (2) the mean; (3) the standard deviation; (4) the median; (5) the lower quartile; (6) the upper quartile; (7) the interquartile range; (8) the maximum value;

[0040] The statistical characteristics of the traffic request time are described as follows: By statistically analyzing the traffic request time of each operation behavior in the entire session, the following can be obtained: (1) mean; (2) standard deviation; (3) median; (4) lower quartile; (5) upper quartile; (6) interquartile range; (7) maximum value;

[0041] The information content measurement of the traffic data is described as follows: by analyzing the operation behavior process in a session from the perspective of information utilization, we can obtain (1) session request rate, which is the proportion of the time when the user requests traffic data from the map tile server in a session to the total session time; (2) session operation rate, which is specifically calculated as 1 - number of user browsing times / number of actual operation behaviors, reflecting the user's operation efficiency in a session; (3) effective tile rate, which is the proportion of the number of traffic that actually constitutes the browsing area in the whole access process to the total number of traffic requested by the user.

[0042] The distribution of traffic data scaling interval levels is described as follows: statistically analyzing the distribution of traffic data requested by users from the map tile service across scaling intervals from 1 to 18.

[0043] Further, the frequency distribution of the operation behavior is described as follows: (1) the frequency of zooming in during the entire session; (2) the frequency of zooming out during the entire session; and (3) the frequency of panning during the entire session.

[0044] Furthermore, the statistical characteristics of user browsing behavior time are described as follows: the time difference between two adjacent seconds of data stream in a session is greater than 2 seconds, which represents the actual browsing time of the user browsing the map. Statistical analysis of the user browsing behavior time can obtain the following user browsing time: (1) mean; (2) standard deviation; (3) median; (4) lower quartile; (5) upper quartile; (6) interquartile range; (7) maximum value;

[0045] The regional characteristics of the user browsing trajectory are described as follows: By statistically analyzing the rectangular translation information during the geographical region discovery process of user operation behavior, we can obtain (1) the total area of ​​the rectangular region of the user browsing area; (2) the maximum value of the column direction of the user browsing area translation; (3) the maximum value of the row direction of the user browsing area translation; (4) the maximum area of ​​the rectangular region of the user browsing area; (5) the size of the average user operation output area; and (6) the number of user browsing areas.

[0046] Further, in step 6, the abnormal behavior training sample dataset obtained in step 4 is divided into a training set, a validation set, and a test set in a 7:2:1 ratio. A machine learning model is built using the LightGBM framework, with the following training parameters: the number of leaf nodes in LightGBM is 63, the objective function is binary cross-entropy, the learning rate is 0.001, the L1 regularization coefficient is 0.1, the L2 regularization coefficient is 0.001, and the number of training rounds is 20,000. An early stopping strategy is set and the model is trained using the training and validation sets. After training, the model stops after N rounds. The training, validation, and test sets are then integrated, the number of training rounds of the model is set to N, the early stopping strategy is canceled, and the model is re-input into the LightGBM model for training. Finally, a machine learning-based crawler behavior detection model for map tile services is obtained.

[0047] Furthermore, in step 7, the session feature vector of the user to be predicted is substituted into the crawler behavior detection model for map tile service based on machine learning to obtain the session detection result of the user to be predicted. If the detection model determines that there is crawler behavior in a certain session of the user, the user is identified as an abnormal user, and the quadruple data of "user id, session id, session start time, session end time" is stored in a spreadsheet. Then, a ban or rate limit policy is implemented on the user.

[0048] This invention proposes an electronic device, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps of the machine learning-based crawler behavior detection method for map tile services.

[0049] This invention proposes a computer-readable storage medium for storing computer instructions, which, when executed by a processor, implement the steps of the machine learning-based crawler behavior detection method for map tile services.

[0050] Compared with existing methods, the method of this invention has two advantages: First, this invention starts from the operating mechanism of crawlers for map tile services, and uses feature engineering to deeply mine the content information of user request traffic data, extracting multi-dimensional map browsing behavior and content features to fully describe crawler behavior; Second, this invention uses the LightGBM framework to build a machine learning model, which on the one hand reduces the large amount of business experience and prior knowledge required for manual strategy design, making it more convenient to apply; on the other hand, machine learning methods can efficiently fit and analyze the features obtained from traffic data mining, ultimately achieving accurate identification of crawler behavior for map tile services.

[0051] The crawler behavior detection method for map tile services based on machine learning proposed in this invention has the following advantages:

[0052] 1. The machine learning-based crawler behavior detection method for map tile services proposed in this invention can detect abnormal crawler behavior with high precision and recall in a real network environment.

[0053] 2. This invention starts from the perspective of the operation mechanism of web crawlers, and uses feature engineering to deeply mine the traffic data generated when users request map service tiles. The established feature vector can fully describe the user's browsing behavior, ensuring the timeliness of the method in the process of fighting against crawler designers.

[0054] 3. This invention constructs a machine learning model based on the LightGBM ensemble learning framework. The model training does not require a large amount of prior knowledge and experience, making the method highly feasible. At the same time, the model also has good nonlinear fitting ability, ensuring the effectiveness of the detection method while also having a certain degree of interpretability.

[0055] 4. This invention performs grouping operations on users and their sessions, and the detection process after grouping can be calculated in parallel, thereby improving computational efficiency. Attached Figure Description

[0056] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0057] Figure 1 This is a flowchart of a map tile data crawler identification method described in the prior art.

[0058] Figure 2 This is a flowchart of the crawler behavior detection method for map tile services based on machine learning proposed in this invention.

[0059] Figure 3 This is a schematic diagram of user session segmentation in this invention.

[0060] Figure 4 This is a schematic diagram illustrating the geographical region discovery of user operation behavior in this invention.

[0061] Figure 5 This is a schematic diagram illustrating the visualization of the user's map browsing trajectory in this invention.

[0062] Figure 6 This is a visualization diagram of the sessions between abnormal crawling behavior and normal behavior in this invention. (a) shows the session display of abnormal crawling behavior, and (b) shows the session display of normal behavior. Detailed Implementation

[0063] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0064] The purpose of this invention is to address the shortcomings of existing web crawler behavior detection methods, such as poor timeliness of detection patterns and insufficient modeling of crawler behavior. This invention proposes a machine learning-based method for detecting web crawler behavior in map tile services. Feature engineering is used to deeply mine traffic data generated by user requests, extracting multi-dimensional map browsing behavior and content features. Then, a machine learning model is built using the LightGBM framework to efficiently fit and analyze the features obtained from the traffic data mining, ultimately achieving accurate identification of web crawler behavior in map tile services. This invention mainly includes seven steps: acquiring traffic data, user browsing session segmentation, reproducing user map browsing trajectory behavior, constructing abnormal behavior training samples, extracting user behavior features, training the abnormal behavior detection model, and identifying and archiving abnormal users. This achieves web crawler behavior detection for map tile services. The technical process is as follows: Figure 2 As shown.

[0065] Specifically, this invention proposes a machine learning-based method for detecting crawler behavior in map tile services, the method comprising the following steps:

[0066] Step 1: Obtain traffic data: This step includes traffic logs obtained through a distributed message queue, file system, or database; each traffic data entry contains the following fields: user ID, request time timestamp, request source IP information, browser information, map tile matrix name, map zoom level, map tile row number, and map tile column number;

[0067] Step 2: User browsing session segmentation: such as Figure 3 As shown, the traffic data obtained in step 1 is divided into multiple data streams, each of which represents the traffic data generated by a user's operation behavior over a continuous period of time.

[0068] Step 2 specifically involves:

[0069] Step 2.1: Group users: Group the traffic data by user ID or user access IP field in the traffic data. Each group of traffic data represents the traffic data generated by all operations of a user.

[0070] Step 2.2: Sort by request time: Sort each group of data in order of seconds according to the request timestamp of the traffic data, and aggregate the traffic data within each second to form a data stream in the form of key-value pairs with "user id, run timestamp" as the key and the set of traffic data per second as the value;

[0071] Step 2.3: Design threshold segmentation browsing sessions: Perform first-order time difference analysis on each data stream, calculate the time interval between each data stream segment and the previous data stream segment, and set the session maintenance interval T. gapIf the time interval between two adjacent data streams is greater than the session maintenance interval, the data is divided using the interval between the two data streams as the boundary. Finally, each group of data streams forms a set of multiple data segments. Each segment is a key-value pair data stream with "user id, session id" as the key and each session traffic data set as the value.

[0072] Step 3: Reproduction of user map browsing trajectory behavior: The data stream after session segmentation is parsed and calculated to obtain the user's map browsing trajectory during the entire session, which is then visualized in the form of a heat map;

[0073] Step 3 specifically involves:

[0074] Step 3.1: parse the traffic data in seconds according to the request time order. Parse each data stream obtained in Step 2 and retain only the map zoom level, map tile row number and map tile column number fields.

[0075] Step 3.2: Determine the start time of the user operation behavior state. Define the user operation behavior as zoom in, zoom out, or pan. Then, sequentially traverse each data stream, comparing the map zoom level of each second-level time window data stream with the map zoom level of the next second-level time window data stream. If the two time window levels are the same, the user operation in the next time window is a pan operation; if the zoom level of the current time window data stream is greater than the zoom level of the next time window, the user operation in the next time window is a zoom out; if the zoom level of the current time window is less than the zoom level of the next second, the user operation in the next time window is a zoom in; if the user operation behavior changes between two adjacent time windows, or if the time interval between two adjacent time windows is greater than 2 seconds, the termination of one operation behavior and the start of another operation behavior can be determined. Mark an operation behavior with "user id, session id, operation behavior type, source map zoom level, target map zoom level, operation behavior start time, operation behavior end time".

[0076] Step 3.3: Geographical region discovery of user operation behavior. For the operation behavior identified in Step 3.2, obtain the data stream from the start time to the end time of the operation behavior. Only retain all data streams under the target map zoom level. The data format is "[(to_z,x_1,y_1),(to_z,x_2,y_2),…]", where to_z is the target map zoom level, x is the map tile column number, and y is the map tile row number. Integrate the data streams to form a two-dimensional array Matrix, and construct the corresponding coordinate mapping function f:(x,y)→(m,n). If the data stream contains data (x,y), then the value of Matrix[f(x,y)] is 1; otherwise, it is 0. Figure 4As shown, after assigning values ​​to the two-dimensional array Matrix, the size of the sliding rectangular window is set (size). x size y The points with a value of 1 in the two-dimensional array are slid and translated to obtain a browsing trajectory of an operation. Each browsing trajectory is marked with "operation, source target map zoom range, target map zoom range, slice coordinates of the lower left corner of the translation matrix, slice coordinates of the upper right corner of the translation matrix, and slice coordinates of the middle of the translation matrix".

[0077] Step 3.4: Integrate user operation behavior paths. Based on steps 3.2 and 3.3, determine all operation behaviors and browsing trajectories in a session, and then store each browsing trajectory in an ordered queue.

[0078] Step 3.5: Coordinate transformation of user operation path. The slice coordinates saved in steps 3.3 and 3.4 are in Mercator coordinate system. The slice coordinates need to be transformed into latitude and longitude coordinate system.

[0079] Step 3.6: Visualize user map browsing trajectory. Design a corresponding front-end interface to display the coordinates of the intermediate slices of the translation matrix of the user operation behavior path obtained in the previous steps in the form of a heatmap. The display result is as follows: Figure 5 As shown;

[0080] Step 4: Constructing Training Samples for Abnormal Behavior: If the visualization of a user's browsing trajectory for a certain session is neatly and densely arranged, forming a large and highly regular rectangular box, then that area represents abnormal data crawling behavior; otherwise, it represents a normal user. Figure 6 (a) shows a session display of abnormal crawling behavior, while Figure 6 (b) Session display of normal map browsing behavior. An abnormal behavior training sample dataset was obtained by manually annotating the browsing trajectories of selected users' session behaviors.

[0081] Step 5: User Behavior Feature Extraction: The features include eight categories: global traffic data features, traffic speed statistics, operation behavior frequency distribution, traffic request time statistics, user browsing behavior time statistics, regional features of user browsing trajectory, traffic data informatics measurement, and traffic data scaling interval level distribution. Each category is described as follows:

[0082] The global characteristics of the traffic data are described as follows: (1) Slice session duration, which refers to the timestamp of the last data stream of a session minus the timestamp of the first data stream; (2) Number of user browsing times, which refers to the number of times the time difference between two adjacent data streams in a session is greater than 2 seconds; (3) Total number of requested traffic data, which refers to the total number of traffic data requests from the user to the map slice server in a session; (4) Number of actual operation behaviors, which refers to the total number of times the user performs zoom-in, zoom-out, and panning operations in a session.

[0083] The traffic speed statistical characteristics are described as follows: (1) the total time spent by users requesting traffic data from the map tile server; (2) the mean; (3) the standard deviation; (4) the median; (5) the lower quartile; (6) the upper quartile; (7) the interquartile range; (8) the maximum value;

[0084] The statistical characteristics of the traffic request time are described as follows: By statistically analyzing the traffic request time of each operation behavior in the entire session, the following can be obtained: (1) mean; (2) standard deviation; (3) median; (4) lower quartile; (5) upper quartile; (6) interquartile range; (7) maximum value;

[0085] The information content measurement of the traffic data is described as follows: by analyzing the operation behavior process in a session from the perspective of information utilization, we can obtain (1) session request rate, which is the proportion of the time when the user requests traffic data from the map tile server in a session to the total session time; (2) session operation rate, which is specifically calculated as 1 - number of user browsing times / number of actual operation behaviors, reflecting the user's operation efficiency in a session; (3) effective tile rate, which is the proportion of the number of traffic that actually constitutes the browsing area in the whole access process to the total number of traffic requested by the user.

[0086] The distribution of traffic data scaling interval levels is described as follows: statistically analyzing the distribution of traffic data requested by users from the map tile service from scaling intervals 1 to 18.

[0087] The frequency distribution of the operation behavior is described as follows: (1) the frequency of zooming in during the entire session; (2) the frequency of zooming out during the entire session; and (3) the frequency of panning during the entire session.

[0088] The statistical characteristics of user browsing behavior time are described as follows: the time difference between two adjacent seconds of data stream in a session is greater than 2 seconds, which represents the actual browsing time of the user browsing the map. Statistical analysis of user browsing behavior time can obtain the following user browsing time: (1) mean; (2) standard deviation; (3) median; (4) lower quartile; (5) upper quartile; (6) interquartile range; (7) maximum value;

[0089] The regional characteristics of the user browsing trajectory are described as follows: By statistically analyzing the rectangular translation information during the geographical region discovery process of user operation behavior, we can obtain (1) the total area of ​​the rectangular region browsed by the user; (2) the maximum value of the column direction of the translation of the user browsing region; (3) the maximum value of the row direction of the translation of the user browsing region; (4) the maximum area of ​​the rectangular region browsed by the user; (5) the size of the average user operation output area; and (6) the number of user browsing regions.

[0090] Step 6: Training the abnormal behavior detection model;

[0091] In step 6, the abnormal behavior training sample dataset obtained in step 4 is divided into training set, validation set, and test set according to a 7:2:1 ratio. A machine learning model is built using the LightGBM framework, with the following training parameters: the number of leaf nodes in LightGBM is 63, the objective function is binary cross-entropy, the learning rate is 0.001, the L1 regularization coefficient is 0.1, the L2 regularization coefficient is 0.001, and the number of training epochs is 20,000. An early stopping strategy is set and the model is trained using the training and validation sets. After training, if the model stops after N epochs, the training set, validation set, and test set are integrated, the number of training epochs of the model is set to N, the early stopping strategy is canceled, and the model is re-input into the LightGBM model for training. Finally, a machine learning-based crawler behavior detection model for map tile services is obtained.

[0092] Step 7: Identify and archive abnormal users.

[0093] In step 7, the session feature vector of the user to be predicted is substituted into the crawler behavior detection model for map tile service based on machine learning to obtain the session detection result of the user to be predicted. If the detection model determines that there is crawler behavior in a certain session of the user, the user is identified as an abnormal user, and the four-tuple data of "user id, session id, session start time, session end time" is stored in a spreadsheet. Then, a ban or rate limit policy is implemented on the user.

[0094] This invention proposes a machine learning-based method for detecting crawler behavior in map tile services, addressing the issues of poor timeliness and low recall in existing crawler behavior detection methods. Taking the Jilin No. 1 network tile package image service as an example, experiments were conducted on a manually labeled abnormal behavior dataset from August 1st to August 31st, 2023, compared with the patent "A Method for Identifying Data Crawlers," with a1=0.2, a2=0.8, b1=0.1, b2=0.9, θ1=6, and θ2=2. The performance on the test set is shown in Table 1, which presents the experimental results of this invention compared to the patent "A Method for Identifying Data Crawlers." Table 1 shows that the detection method proposed in this invention has a detection accuracy of 100% and a recall rate of 90%, significantly better than the comparison method's accuracy of 87.5% and recall rate of 40%. Experimental results show that, compared with traditional detection methods for map tile services that utilize rule-based strategies, this invention can detect abnormal behavior more accurately and comprehensively with higher precision and recall, resulting in a significant improvement in performance indicators. Furthermore, because this invention starts from the perspective of web crawler operation mechanism and mines various user behavior features from traffic data, it leverages the powerful fitting capabilities of machine learning to ensure the timeliness of the method in the process of confronting crawler designers.

[0095] Table 1 shows the experimental results of this invention compared to the patent "A Method for Identifying Data Crawlers".

[0096]

[0097] This invention proposes an electronic device, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps of the machine learning-based crawler behavior detection method for map tile services.

[0098] This invention proposes a computer-readable storage medium for storing computer instructions, which, when executed by a processor, implement the steps of the machine learning-based crawler behavior detection method for map tile services.

[0099] The memory in this application embodiment can be volatile memory or non-volatile memory, or it can include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct rambus RAM (DRRAM). It should be noted that the memory used in the methods described in this invention is intended to include, but is not limited to, these and any other suitable types of memory.

[0100] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., high-density digital video discs (DVDs)), or semiconductor media (e.g., solid-state disks (SSDs)).

[0101] In implementation, each step of the above method can be completed by integrated logic circuits in the processor's hardware or by instructions in software. The steps of the method disclosed in the embodiments of this application can be directly implemented by a hardware processor, or by a combination of hardware and software modules in the processor. The software modules can reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method. To avoid repetition, detailed descriptions are omitted here.

[0102] It should be noted that the processor in the embodiments of this application can be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method embodiments can be completed by the integrated logic circuitry in the processor's hardware or by instructions in software form. The processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied as being executed by a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. This storage medium is located in memory, and the processor reads the information in the memory and, in conjunction with its hardware, completes the steps of the above methods.

[0103] The above provides a detailed description of the crawler behavior detection method for map tile services based on machine learning proposed in this invention. Specific examples have been used to illustrate the principles and implementation methods of this invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this invention. Therefore, the content of this specification should not be construed as a limitation of this invention.

Claims

1. A machine learning-based method for detecting crawler behavior in map tile services, characterized in that, The method includes the following steps: Step 1: Obtain traffic data: This step includes traffic logs obtained through a distributed message queue, file system, or database; each traffic data entry contains the following fields: user ID, request time timestamp, request source IP information, browser information, map tile matrix name, map zoom level, map tile row number, and map tile column number; Step 2: User browsing session segmentation: The traffic data obtained in Step 1 is segmented into multiple data streams. Each data stream represents the traffic data generated by a user's operation behavior over a continuous period of time. Step 3: Reproduction of user map browsing trajectory behavior: The data stream after session segmentation is parsed and calculated to obtain the user's map browsing trajectory during the entire session, which is then visualized in the form of a heat map; Step 4: Construction of abnormal behavior training samples: If the visualization of a user's browsing trajectory for a certain session is neatly and densely arranged, forming a large and very regular rectangle, then that area represents abnormal data crawling behavior; otherwise, it represents a normal user. By manually selecting a portion of users' session browsing trajectories and manually annotating them, an abnormal behavior training sample dataset is obtained. Step 5: User behavior feature extraction: The features include global traffic data features, traffic speed statistics features, operation behavior frequency distribution, traffic request time statistics features, user browsing behavior time statistics features, regional features of user browsing trajectory, traffic data information content measurement, and traffic data scaling interval level distribution, totaling 8 categories of features. Step 6: Training the Abnormal Behavior Detection Model; In Step 6, the abnormal behavior training sample dataset obtained in Step 4 is divided into a training set, a validation set, and a test set in a 7:2:1 ratio; a machine learning model is built using the LightGBM framework, with the following training parameters: the number of leaf nodes in LightGBM is 63, the objective function is binary cross-entropy, the learning rate is 0.001, the L1 regularization coefficient is 0.1, the L2 regularization coefficient is 0.001, the number of training epochs is 20,000, an early stopping strategy is set, and the model is trained using the training and validation sets; after training, if the model stops after N epochs, the training, validation, and test sets are integrated, the number of training epochs of the model is set to N, the early stopping strategy is canceled, and the model is re-input into the LightGBM model for training, finally obtaining a machine learning-based crawler behavior detection model for map tile services; Step 7: Identify and archive abnormal users.

2. The method according to claim 1, characterized in that, Step 2 specifically involves: Step 2.1: Group users: Group the traffic data by user ID or user access IP field in the traffic data. Each group of traffic data represents the traffic data generated by all operations of a user. Step 2.2: Sort by request time: Sort each group of data in order of seconds according to the request timestamp of the traffic data, and aggregate the traffic data within each second to form a data stream in the form of key-value pairs with "user id, run timestamp" as the key and the set of traffic data per second as the value; Step 2.3: Design threshold segmentation browsing sessions: Perform first-order time difference analysis on each data stream, calculate the time interval between each data stream segment and the previous data stream segment, and set the session maintenance interval. If the time interval between two adjacent data streams is greater than the session maintenance interval, the data is divided using the interval between the two data streams as the boundary. Ultimately, each group of data streams forms a set of multiple data segments. Each segment is a key-value pair data stream with "user id, session id" as the key and each session traffic data set as the value.

3. The method according to claim 1, characterized in that, Step 3 specifically involves: Step 3.1: parse the traffic data in seconds according to the request time order. Parse each data stream obtained in Step 2 and retain only the map zoom level, map tile row number and map tile column number fields. Step 3.2: Determine the start time of the user operation behavior state. Define the user operation behavior as zoom in, zoom out, or pan. Then, sequentially traverse each data stream, comparing the map zoom level of each second-level time window data stream with the map zoom level of the next second-level time window data stream. If the two time window levels are the same, the user operation in the next time window is a pan operation; if the zoom level of the current time window data stream is greater than the zoom level of the next time window, the user operation in the next time window is a zoom out; if the zoom level of the current time window is less than the zoom level of the next second, the user operation in the next time window is a zoom in; if the user operation behavior changes between two adjacent time windows, or if the time interval between two adjacent time windows is greater than 2 seconds, the termination of one operation behavior and the start of another operation behavior can be determined. Mark an operation behavior with "user id, session id, operation behavior type, source map zoom level, target map zoom level, operation behavior start time, operation behavior end time". Step 3.3: Geographical region discovery of user operation behavior. For the operation behavior identified in Step 3.2, obtain the data stream from the start time to the end time of the operation behavior. Only retain all data streams under the target map zoom level. The data format is "[(to_z, x_1, y_1), (to_z, x_2, y_2), …]", where to_z is the target map zoom level, x is the map tile column number, and y is the map tile row number. Integrate the data streams to form a two-dimensional array Matrix, and construct the corresponding coordinate mapping function. If data exists in the data stream Then Matrix[ The value of ] is 1 if it is not 0 otherwise; after assigning values ​​to the two-dimensional array Matrix, the size of the sliding rectangular window is set. The points with a value of 1 in the two-dimensional array are slid and translated to obtain a browsing trajectory of an operation. Each browsing trajectory is marked with "operation, source target map zoom range, target map zoom range, slice coordinates of the lower left corner of the translation matrix, slice coordinates of the upper right corner of the translation matrix, and slice coordinates of the middle of the translation matrix". Step 3.4: Integrate user operation behavior paths. Based on steps 3.2 and 3.3, determine all operation behaviors and browsing trajectories in a session, and then store each browsing trajectory in an ordered queue. Step 3.5: Coordinate transformation of user operation path. The slice coordinates saved in steps 3.3 and 3.4 are in Mercator coordinate system. The slice coordinates need to be transformed into latitude and longitude coordinate system. Step 3.6: Visualize user map browsing trajectory. Design the corresponding front-end interface to display the coordinates of the intermediate slices of the translation matrix of the user operation behavior path obtained in the previous steps in the form of a heat map.

4. The method according to claim 1, characterized in that, The global characteristics of the traffic data are described as follows: (1) Slice session duration, which refers to the timestamp of the last data stream of a session minus the timestamp of the first data stream; (2) Number of user browsing times, which refers to the number of times the time difference between two adjacent data streams in a session is greater than 2 seconds; (3) Total number of requested traffic data, which refers to the total number of traffic data requests from the user to the map slice server in a session; (4) Number of actual operation behaviors, which refers to the total number of times the user performs zoom-in, zoom-out, and panning operations in a session. The traffic speed statistical characteristics are described as follows: (1) the total time spent by the user requesting traffic data from the map tile server; (2) the mean; (3) the standard deviation; (4) the median; (5) the lower quartile; (6) the upper quartile; (7) the interquartile range; (8) the maximum value; The statistical characteristics of the traffic request time are described as follows: By statistically analyzing the traffic request time of each operation behavior in the entire session, the following can be obtained: (1) mean; (2) standard deviation; (3) median; (4) lower quartile; (5) upper quartile; (6) interquartile range; (7) maximum value; The information content measurement of the traffic data is described as follows: by analyzing the operation behavior process in a session from the perspective of information utilization, we can obtain (1) session request rate, which is the proportion of the time when the user requests traffic data from the map tile server in a session to the total session time; (2) session operation rate, which is specifically calculated as 1 - number of user browsing times / number of actual operation behaviors, reflecting the user's operation efficiency in a session; (3) effective tile rate, which is the proportion of the number of traffic that actually constitutes the browsing area in the whole access process to the total number of traffic requested by the user. The distribution of traffic data scaling interval levels is described as follows: statistically analyzing the distribution of traffic data requested by users from the map tile service across scaling intervals from 1 to 18.

5. The method according to claim 1, characterized in that, The frequency distribution of the operation behavior is described as follows: (1) the frequency of zooming in during the whole session; (2) the frequency of zooming out during the whole session; and (3) the frequency of panning during the whole session.

6. The method according to claim 1, characterized in that, The statistical characteristics of user browsing behavior time are described as follows: the time difference between two adjacent seconds of data stream in a session is greater than 2 seconds, which represents the actual browsing time of the user browsing the map. Statistical analysis of user browsing behavior time can obtain the following user browsing time: (1) mean; (2) standard deviation; (3) median; (4) lower quartile; (5) upper quartile; (6) interquartile range; (7) maximum value; The regional characteristics of the user browsing trajectory are described as follows: By statistically analyzing the rectangular translation information during the geographical region discovery process of user operation behavior, we can obtain (1) the total area of ​​the rectangular region of the user browsing area; (2) the maximum value of the column direction of the user browsing area translation; (3) the maximum value of the row direction of the user browsing area translation; (4) the maximum area of ​​the rectangular region of the user browsing area; (5) the size of the average user operation output area; and (6) the number of user browsing areas.

7. The method according to claim 1, characterized in that, In step 7, the session feature vector of the user to be predicted is substituted into the crawler behavior detection model for map tile service based on machine learning to obtain the session detection result of the user to be predicted. If the detection model determines that there is crawler behavior in a certain session of the user, the user is identified as an abnormal user, and the four-tuple data of "user id, session id, session start time, session end time" is stored in a spreadsheet. Then, a ban or rate limit policy is implemented on the user.

8. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1-7.

9. A computer-readable storage medium for storing computer instructions, characterized in that, When the computer instructions are executed by the processor, they implement the steps of the method according to any one of claims 1-7.

Images