Sample labeling method, device, and readable storage medium

Abstract

This application discloses a sample annotation method, device, and readable storage medium. The platform server acquires a dataset containing multiple access logs to be annotated. Based on the IP address and domain name of each access log, the dataset is divided into multiple groups, with different IP addresses or domain names for the access logs in different groups. For each group, the access logs are annotated according to the constraint parameters contained in each access log within that group, thus obtaining a training dataset for training a web crawler detection model. Using this approach, the platform server automatically determines and annotates positive and negative samples within groups based on different constraint parameters and combinations thereof, freeing it from the subjective influence of professional annotators. This minimizes the probability of mislabeling and missed labeling, efficiently and effectively annotating massive amounts of access logs while ensuring the consistency of the annotated data, achieving the goal of quickly and accurately obtaining a high-quality training dataset.

Description

Technical Field

[0001] This application relates to the field of network security technology, and in particular to a sample annotation method, device, and readable storage medium. Background Technology

[0002] A web crawler, also known as a web spider, is a program or script that automatically retrieves information from the web according to certain rules. Some criminals use web crawlers for malicious crawling operations.

[0003] To identify malicious web crawlers, a crawler detection model is typically trained to identify traffic on target websites. The training process involves acquiring a large amount of sample data to form a sample set, which is then used to train an initial model, resulting in the crawler detection model. The sample set includes both positive and negative samples.

[0004] Typically, the performance and accuracy of a web crawler detection model depend on the quality and quantity of labeled data in the sample data. Quantity refers to the total number of positive and negative samples, while quality is usually related to the proportion of mislabeled samples. Mislabeled samples include positive samples that are incorrectly labeled as negative samples and negative samples that are incorrectly labeled as positive samples.

[0005] Currently, sample data is obtained by professional personnel annotating access logs. However, due to the subjective influence of the annotators, annotation errors are introduced, resulting in low performance and poor accuracy of the recognition model trained based on this annotated data. Summary of the Invention

[0006] This application provides a sample annotation method, device, and readable storage medium. It automatically analyzes access logs corresponding to crawler traffic from massive logs, minimizing the probability of mislabeling and achieving efficient annotation of massive data while ensuring the accuracy of the training dataset.

[0007] In a first aspect, embodiments of this application provide a sample annotation method, including:

[0008] Obtain the dataset to be labeled, which contains multiple access logs to be labeled, and the access logs correspond one-to-one with the access requests initiated by the client to access the target website;

[0009] The dataset to be labeled is divided into multiple groups based on the IP address and domain name of each access log. The IP address or domain name of the access logs in different groups are different.

[0010] For each of the plurality of groups, each access log in the group is labeled according to the constraint parameters contained in each access log in the group to obtain a training dataset for training the crawler detection model.

[0011] Secondly, embodiments of this application provide a sample labeling device, comprising:

[0012] The acquisition module is used to acquire the dataset to be labeled. The dataset to be labeled contains multiple access logs to be labeled. The access logs correspond one-to-one with the access requests initiated by the client to access the target website.

[0013] The processing module is used to divide the dataset to be labeled according to the IP address and domain name of each access log to obtain multiple groups, and the IP address or domain name of the access logs in different groups are different.

[0014] The annotation module is used to annotate each access log in the group according to the constraint parameters contained in each access log in the group for each of the multiple groups, so as to obtain a training dataset for training the crawler detection model.

[0015] Thirdly, embodiments of this application provide an electronic device, including: a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the electronic device implements the method described in various possible implementations of the first aspect above.

[0016] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer instructions, which, when executed by a processor, are used to implement the methods described in various possible implementations of the first aspect above.

[0017] The sample annotation method, device, and readable storage medium provided in this application involve a platform server acquiring a dataset of access logs to be annotated, containing multiple access logs to be annotated. The dataset is then divided into multiple groups based on the IP address and domain name of each access log, with different IP addresses or domain names for the access logs in different groups. Then, for each group, each access log in the group is annotated according to the constraint parameters contained therein, resulting in a training dataset for training a crawler detection model. Using this approach, the platform server automatically determines and annotates positive and negative samples in each group based on different constraint parameters and combinations thereof, free from the subjective influence of professional annotators. This minimizes the probability of mislabeling and missed labeling, efficiently and effectively annotating massive amounts of access logs while ensuring the consistency of the annotated data, thus achieving the goal of quickly and accurately obtaining a high-quality training dataset. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a schematic diagram of the network architecture of the sample annotation method provided in the embodiments of this application;

[0020] Figure 2 This is a flowchart of the sample annotation method provided in the embodiments of this application;

[0021] Figure 3 This is a schematic diagram of the sample annotation method provided in the embodiments of this application;

[0022] Figure 4 A schematic diagram of a sample labeling device provided in an embodiment of this application;

[0023] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0024] Supervised learning refers to the process of training a model using a large amount of labeled sample data, and then having that model produce corresponding outputs based on the inputs. Data labeling is the lifeblood of supervised learning models; the performance and accuracy of the model depend on the quality and quantity of the labeled data.

[0025] A web crawler detection model refers to a model obtained by training an initial model using a large number of labeled access logs. Labeling access logs requires labelers to have certain log auditing and event reconstruction capabilities. Therefore, labelers are often professionals in their respective fields, and such professionals are extremely scarce, resulting in high labeling thresholds and costs.

[0026] Moreover, even when professional annotators label the samples, some drawbacks are unavoidable. These are described in detail below:

[0027] 1. It is difficult to control the consistency of labeled data.

[0028] The annotation task is heavily influenced by the subjectivity of the annotators and reviewers, which can introduce annotation errors. Furthermore, annotators need to perform tedious analysis of access logs in conjunction with the client's actual business needs, leading to review fatigue and errors. Additionally, the sheer volume of annotated data makes it difficult to re-examine the data.

[0029] 2. The quality and speed of labeling are difficult to guarantee.

[0030] Massive access logs are characterized by their large quantity, wide range of dimensions, and complex data types, posing a significant challenge to data annotation and requiring professional annotators to possess certain log review and event reconstruction capabilities.

[0031] 3. Access logs that are difficult to detect abnormalities.

[0032] As the battle between web crawlers and anti-crawler measures intensifies, some low-level web crawlers are easily detected by conventional detection methods. However, advanced web crawlers, in order to bypass various anti-crawler measures, will try their best to disguise themselves as normal users, and their access characteristics are almost indistinguishable from those of normal users. Even professional annotators cannot directly determine whether the access is abnormal, which makes it difficult to accurately annotate access logs.

[0033] Low-level crawlers refer to crawlers with simple behaviors and clearly defined characteristics. For example, when a low-level crawler visits website A, assuming website A has a pagination function, the low-level crawler will visit dozens of different pages in a very short time, such as within one second, which is obviously impossible for a normal user. Clearly defined characteristics mean that the low-level crawler's request headers carry keywords such as curl, requests, and python. When these keywords appear in the access logs, the crawler is marked as a negative sample.

[0034] Advanced web crawlers exhibit access behavior almost identical to that of normal users, and the characteristic distribution in their access logs closely resembles that of normal users. For example, on a flight ticket purchase website, a normal user's access sequence is: search for tickets → vote for tickets → purchase tickets. A low-level crawler might skip the ticket search and voting steps and directly purchase and pay for tickets. Advanced web crawlers, however, utilize automation frameworks and other tools to completely simulate user purchasing behavior, following the normal process.

[0035] 4. Different customer types have different access characteristics, which can easily lead to mislabeling or omission of labels.

[0036] As mentioned above, even professional annotators struggle to quickly obtain consistent, high-quality annotation data. Furthermore, the inability to recognize access logs from advanced web crawlers and the inability to cater to different client types easily leads to missed or incorrect annotations.

[0037] Based on this, embodiments of this application provide a sample annotation method, device, and readable storage medium, which automatically analyzes access logs corresponding to crawler traffic from massive logs, minimizing the probability of mislabeling, ensuring the consistency of labeled data, and achieving efficient annotation of massive data.

[0038] Figure 1 This is a schematic diagram of the network architecture of the sample annotation method provided in the embodiments of this application. Please refer to it. Figure 1The network architecture includes a platform server 11, a web server 12, and a terminal device 13. A network connection is established between the platform server 11 and the web server 12, and a network connection is established between the platform server 11 and the terminal device 13.

[0039] Platform server 11, for example, is a Software-as-a-Service (SASS) platform. Access requests initiated by users through terminal device 13 to access website server 12 arrive at website server 12 after passing through platform server 11. Similarly, response messages from website server 12 arrive at terminal device 13 after passing through platform server 11. A crawler detection model is deployed on platform server 11 to detect and process access requests initiated by crawler programs.

[0040] The crawler detection model is obtained by labeling access logs over a period of time on the platform server 11 to obtain a training dataset, and then using the training dataset to train the initial model.

[0041] Website server 12 is the server of the target website, and can be either hardware or software. When website server 12 is hardware, it can be a single server or a distributed server cluster composed of multiple servers. When website server 12 is software, it can be multiple software modules or a single software module, etc., and this application embodiment is not limited.

[0042] The terminal device 13 can be either hardware or software. When the terminal device 13 is hardware, it can be, for example, a mobile phone, tablet computer, e-book reader, laptop computer, desktop computer, server, etc. When the terminal device 13 is software, it can be installed in the hardware devices listed above. In this case, the terminal device 13 can be, for example, multiple software modules or a single software module, etc., and the embodiments of this application are not limited.

[0043] It should be understood that, Figure 1 The number of platform servers 11, web servers 12, and terminal devices 13 shown is merely illustrative. In actual implementation, any number of platform servers 11, web servers 12, and terminal devices 13 can be deployed according to actual needs.

[0044] It should be noted that, although Figure 1In the network architecture shown, the platform server 11 generates and stores access logs, and annotates the access logs. However, this embodiment is not limited to this. In other feasible implementations, for each access request from the terminal device 13, the platform server 11 generates and stores the access log corresponding to that access request. Other servers specifically used for training the model obtain these access logs, annotate the access logs to obtain a training dataset, train a crawler detection model, and deploy it on the platform server 11.

[0045] Below, based on Figure 1 The network architecture shown illustrates the sample annotation method provided in this application embodiment. For examples, please refer to... Figure 2 .

[0046] Figure 2 This is a flowchart of the sample annotation method provided in this application embodiment. This embodiment is described from the perspective of the platform server. This embodiment includes:

[0047] 201. Obtain the dataset to be labeled. The dataset to be labeled contains multiple access logs to be labeled. Each access log corresponds one-to-one with an access request initiated by the client to access the target website.

[0048] Please refer to Figure 1 Every access request from a terminal device passes through the platform server. Similarly, every response message from the website server also passes through the platform server. For each pair of access request and response messages, the platform server generates and stores an access log. For simplicity, the access log for a pair of access request and response messages will be referred to as the access log corresponding to the access request, with a one-to-one correspondence between access requests and access logs. The access log records the initiation time of the access request, the values ​​of each request header in the access request, the values ​​of the response headers in the response message, and whether the access request triggered existing crawler protection rules, etc. The platform server generates and stores an access log for each access request.

[0049] When training a model, if a training dataset is needed, the platform server obtains access logs over a period of time, such as access logs from the past 10 minutes, 15 minutes, 1 hour, and 24 hours of access to the target website. These access logs are used as the access logs to be labeled, thus obtaining the dataset to be labeled.

[0050] 202. Based on the IP address and domain name of each access log, divide the dataset to be labeled to obtain multiple groups, with different IP addresses or domain names for the access logs in different groups.

[0051] Each access log contains the IP address of the terminal device that initiated the access request, the URL in the access request, etc. The platform server extracts the IP address and domain name of each access log. Then, the dataset to be labeled is grouped at the granularity of IP address and domain name, thus dividing the dataset into multiple groups. Access logs in different groups have different IP addresses or domain names. For example, if group 1 and group 2 are two different groups, then group 1 and group 2 have different IP addresses and different domain names; or, group 1 and group 2 have the same IP address and different domain names; or, group 1 and group 2 have different IP addresses and the same domain name.

[0052] By grouping, we can obtain all access information for each IP address to a domain name within a certain time period, which can be reflected through constraint parameters. Constraint parameters include, but are not limited to, IP address, attack type, access path, access time, status code, User-Agent value, Referer value, CID, etc.

[0053] 203. For each of the plurality of groups, label each access log in the group according to the constraint parameters contained in each access log in the group, so as to obtain a training dataset for training the crawler detection model.

[0054] After grouping, the platform server uses IP and domain name as the main components. For each group, based on the constraint parameters contained in the access logs of that group, it determines whether there is abnormal access behavior for the IP corresponding to that group. This automatically identifies the access logs corresponding to access requests initiated by crawlers and access logs corresponding to access requests initiated by normal users. Access logs corresponding to access requests initiated by crawlers are marked as negative samples, and access logs corresponding to access requests initiated by normal users are marked as positive samples. Positive samples are also called white samples, and negative samples are also called black samples.

[0055] The constraint parameters refer to the parameters recorded in the access logs, including but not limited to URLs, User-Agent values, and status codes. Different constraint parameters represent different dimensions. The platform server extracts these multi-dimensional constraint parameters and determines whether an IP exhibits abnormal access behavior based on multiple dimensions or combinations thereof. For example, the platform server determines whether an IP is abnormal based on the User-Agent value of each access log in a group; alternatively, it determines whether an IP is abnormal based on the status code of each access log in a group; still another example is that the platform server combines the User-Agent value and the status code to determine whether an IP is abnormal. If an IP in a certain group is abnormal, it indicates that the IP belongs to the terminal device where the crawler program is located. If an IP in a certain group is normal, it indicates that the IP belongs to the terminal device of a normal user.

[0056] During the annotation process of each access log in the dataset to be labeled, if all access logs in a group are negative samples, an additional field is added to record the rules that the group hits, which facilitates review and continuous improvement of the rules, forming a closed loop.

[0057] The sample annotation method provided in this application involves a platform server acquiring a dataset of access logs to be annotated, containing multiple logs. The dataset is then divided into multiple groups based on the IP address and domain name of each access log, with different IP addresses or domain names for the access logs in different groups. For each group, the access logs within that group are annotated to obtain a training dataset for training a web crawler detection model. This approach automatically determines and annotates positive and negative samples within groups based on different constraint parameters and combinations thereof, eliminating the influence of subjective factors from professional annotators. This minimizes the probability of mislabeling and omissions, efficiently and effectively annotating massive amounts of access logs while ensuring data consistency, ultimately achieving the goal of quickly and accurately obtaining a high-quality training dataset.

[0058] Optionally, in the above embodiments, during the process of the platform server labeling each access log in the group according to the constraint parameters contained in each access log in the group, firstly, for each of the plurality of groups, the service type of the group is determined according to the constraint parameters contained in each access log in the group. Then, each access log in the group is labeled according to the service type of the group.

[0059] For example, different customer types have different access characteristics. If a uniform set of rules is used for labeling, it is very likely to lead to omissions or mislabeling because the rules are not suitable. Therefore, for each group, the platform server determines the business type based on the URL and other constraint parameters contained in the access logs of that group, and then labels each access log according to the business type.

[0060] This approach differentiates between business types and labels each access log in the group according to the business type, thus taking into account the different business types of different customers, avoiding mislabeling and omissions, and improving the accuracy of the training dataset.

[0061] Optionally, in the above embodiments, during the process of determining the business type, the constraint parameter is, for example, a URL. For each of the multiple groups, the platform server extracts a Uniform Resource Locator (URL) from the access logs contained in the group to obtain multiple URLs. Then, the platform server determines a first type of URL and a second type of URL from the multiple URLs. When the proportion of the first type of URL among the multiple URLs is less than a preset threshold, the business type of the group is determined to be an API business; when the proportion of the first type of URL among the multiple URLs is greater than or equal to the preset threshold, the business type of the group is determined to be a WEB business. Here, the first type of URL is a URL that loads static resources, and the second type of URL is a URL that does not require loading static resources.

[0062] For example, please refer to Figure 1 Website servers include web-type sites and pure application programming interface (API) sites, i.e., pure API sites. For clarity, the services provided by web-type sites will be referred to as web services, and the services provided by pure API sites will be referred to as API services.

[0063] In this embodiment, access requests to web-based websites often load static resources such as images, audio, video, scripts (JavaScript, JS), or Cascading Style Sheets (CSS). Additionally, some web-based website access requests also include API access requests. Access logs record loaded static resources, requested APIs, etc.

[0064] The platform server determines whether an access request requests the loading of static resources by analyzing the URL's suffix. If a URL requests the loading of static resources, the suffix includes, but is not limited to, .js, .html, .png, .jpg, etc.

[0065] For example, if a terminal device requests access to the homepage of a target website, the access log will record the following information:

[0066] Visit the homepage: www.xxx.com;

[0067] Load font: www.xxx.com / font.css

[0068] Load JS: www.xxx.com / index.js

[0069] Get the city list via API: www.xxx.com / getCityList /

[0070] Loading homepage image: www.xxx.com / index.png

[0071] Clearly, if the URLs of a group contain requests for accessing API interfaces, then the business corresponding to that group could be either a web business or an API business. This is because web business access requests can also request access to API interfaces.

[0072] Access requests to a pure API site only access the API interface and there is no need to load static resources. Access logs only record the API interfaces accessed.

[0073] After the platform server extracts the URLs from the access logs of a group, it identifies the URLs that require loading static resources (i.e., the first type of URLs) and the URLs that do not require loading static resources (i.e., the second type of URLs). Then, the platform server determines the proportion of the first type of URLs. If the proportion of the first type of URLs is less than a preset threshold, the group's business type is considered an API service; if the proportion of the first type of URLs is greater than or equal to the preset threshold, the group's business type is considered a web service. The first threshold can be adjusted according to the specific web service, etc., and is not limited in this embodiment.

[0074] This approach distinguishes between API services and WEB services, achieving compatibility with different customer business types.

[0075] Optionally, in the above embodiments, when the business type of a group is a web service, the platform server determines the access sequence based on multiple URLs of the group. Then, it determines the information entropy of the access sequence and labels each access log in the group according to the information entropy.

[0076] For example, for a given group, the platform server extracts URLs from the group's access logs, obtaining multiple URLs. These URLs are used to determine the group's business type. Furthermore, if the group's business type is web-based, the platform server further determines the access sequence based on these URLs, calculates information entropy, and labels each access log. This is because, compared to the disordered access behavior of normal users, web crawlers generally have targeted and regular access patterns. For example, a crawler might centrally access the required URLs, resulting in a significant difference in information entropy between the crawler's access sequence and that of a normal user. By calculating the information entropy of the group's access sequence and comparing it with a preset information entropy threshold, it can be determined whether the traffic of a group is abnormal.

[0077] Therefore, if it is impossible to determine whether the access log of a group is a normal traffic access log or an abnormal traffic access log, information entropy can be used to determine it.

[0078] The platform server determines the access sequence based on multiple URLs in a group. This access sequence is actually several URLs arranged in order. For example, if a user buys a plane ticket on an airline website, the actions are: opening the homepage, searching for tickets, and purchasing tickets. The URLs of the homepage, the ticket search page, and the ticket purchase page form an access sequence. The access sequence may also include API requests, etc.

[0079] After determining the access sequence, the platform server determines the information entropy of the access sequence. The platform server can calculate the information entropy using the following formula (1).

[0080]

[0081] Where p(x) i ) represents a random event X = x. i The probability of an event is such that, for a given event, the lower the probability of occurrence, the greater the information content; conversely, the higher the probability of occurrence, the smaller the information content, with a minimum value not less than 0. n represents the number of URLs in the access sequence, and x... i This represents a URL within an access sequence. For example, if the access sequence is ABABABABABA, where A and B represent different URLs, this sequence is a low-probability event compared to normal access, resulting in higher information entropy. Conversely, if the access sequence is ABCDFGGH, this sequence is a high-probability event, resulting in lower information entropy.

[0082] After the platform server determines the information entropy, if the information entropy is less than the preset information entropy, then each access log of that group is marked as a negative sample. If the information entropy is greater than or equal to the preset information entropy, then each access log of that group is marked as a positive sample. The preset information entropy is, for example, 4; the information entropy of a normal access sequence is, for example, 5; and the information entropy of an abnormal access sequence is, for example, 1 or 2, etc.

[0083] This approach determines the access sequence based on multiple URLs within a group, and then labels the access logs contained in the group based on the information entropy of the access sequence, thereby achieving the goal of accurately labeling access logs.

[0084] Optionally, in the above embodiments, for each of the plurality of groups, during the process of the platform server annotating each access log in the group according to the constraint parameters contained in each access log in the group, the constraint parameters are extracted from each access log in the group, and each access log in the group is annotated according to the constraint parameters. The constraint parameters include at least one of the following parameters: the value of User-Agent, the value of Referer, a status code, an attack type, or a client unique identifier (CID), wherein the attack type is used to indicate the protection measures triggered by the access request corresponding to the access log.

[0085] For example, an access log may contain multiple constraint parameters, which the platform server extracts from the access log. These constraint parameters include, but are not limited to, the following:

[0086] a) The value of User-Agent.

[0087] User-Agent, or UA for short, is a common request header that carries information such as the browser type, operating system and version, CPU type, browser rendering engine, browser language, and browser plugins of the terminal device. Since most web crawlers forge the UA value when accessing target websites, crawler traffic can be determined by checking whether the UA value is forged and whether the terminal device's browser is a legitimate browser.

[0088] b) The value of Referer.

[0089] Referer is also a common request header used to record the previous hop source of this access. When the previous hops of an IP are all empty or highly concentrated, and the access volume is very large, the IP is considered to be the IP of the web crawler.

[0090] c) Status code

[0091] After receiving an access request, the target website sends a response message to the terminal device, which carries a response code. The status codes of uncontrolled or malicious web crawlers differ significantly from those of normal access; malicious crawlers exhibit a large number of abnormal status codes. Common status codes include: information response status codes (100-199), success response status codes (200-299), redirection response status codes (300-399), client error response status codes (400-499), and server error response status codes (500-599). Web crawlers also generate a large number of 3xx, 4xx, and 5xx error response status codes. For example, a large number of 3xx status codes likely indicates that the crawler attempted to brute-force the user's password on the login page, failed, and was redirected to the login page, resulting in a large number of 3xx codes.

[0092] d) Attack Type

[0093] The attack type is a result calculated by the platform server's internal engine and is used to indicate the protective measures triggered by the access request corresponding to the access log. If a protective measure has already been triggered for an IP address, the attack type also indicates which protective measure has been triggered.

[0094] e) Client unique identifier (client ID, CID)

[0095] The platform server sends a piece of code to the website server to calculate the CID. Upon receiving an access request from a terminal device, the website server uses this code to calculate the CID based on the terminal device's screen resolution, mouse movements, etc., and then returns it to the terminal device. Each subsequent access to the target website by the terminal device will include this CID. If an access log does not contain a CID, it indicates an abnormal access request. Furthermore, if a packet contains multiple different CIDs, it means that the IP address was used by multiple devices to complete a single access within a short period, making it an abnormal IP address.

[0096] In this embodiment of the application, the platform server can flexibly label access logs according to constraint parameters.

[0097] In one approach, different constraint parameters represent different dimensions, and the platform server labels access logs based on constraint parameters of a single dimension. For example, the platform server labels access logs based on the User-Agent value; another example is labeling access logs based on the status code; yet another example is labeling access logs based on the Referer value.

[0098] In another approach, the platform server annotates access logs using multi-dimensional constraint parameters. The platform server can freely combine rules from various dimensions based on actual business needs, thus forming more precise rules. For example, the platform server might annotate access logs based on the User-Agent and Referer values; or it might annotate access logs based on status codes and attack types.

[0099] In addition, time difference is also one of the important differences between crawler programs and normal access. The time parameters in the logs of crawler programs have a certain regularity and periodicity. Therefore, samples can be labeled in combination with the time dimension to enhance the accuracy and interpretability of the training dataset.

[0100] This approach uses IP addresses as the primary data source, combined with multiple dimensions and algorithms to label access logs, automatically classifying web crawler traffic. This reduces the probability of mislabeling while ensuring efficient labeling of massive amounts of data and guaranteeing the accuracy of the training dataset.

[0101] Optionally, in the above embodiments, when the constraint parameters include User-Agent values, during the process of labeling each access log in the group according to the constraint parameters, the platform server first determines a first quantity, a first proportion, and a second proportion. Then, if the group satisfies at least one UA condition, each access log in the group is labeled as a negative sample; if the group does not satisfy each UA condition, each access log in the group is labeled as a positive sample. Here, the first quantity is the number of unique User-Agent values ​​in the group, the first proportion is the proportion of access logs in the group containing abnormal User-Agent values, and the second proportion is the proportion of access logs in the group containing User-Agent values ​​with a length greater than a first length or less than a second length. The at least one UA condition includes: the first proportion being greater than a preset quantity, the first proportion being greater than a first threshold, and the second proportion being greater than a second threshold.

[0102] For example, User-Agent (UA) is a common request header that carries information such as the terminal device's browser type, operating system and version, CPU type, browser rendering engine, browser language, and browser plugins. For each group, the platform server extracts the UA value from each access log and then labels the access logs according to the UA value.

[0103] In one approach, the platform server deduplicates the User Agent (UA) values, resulting in a predefined count of UA values. If this count exceeds a preset limit, the platform server considers the IP address corresponding to that group to be frequently switching UAs, indicating that the traffic from that group is crawler traffic. Therefore, the platform server marks the access logs for that group as negative samples. If the predefined count is less than or equal to the preset limit, the platform server considers the IP address corresponding to that group to be not frequently switching UAs, indicating that the traffic from that group is normal traffic. Therefore, the platform server marks the access logs for that group as positive samples.

[0104] In another approach, the platform server determines whether the User-Agent value in each access log is normal and determines the proportion of access logs containing abnormal User-Agent values; this proportion is called the first proportion. Abnormal User-Agent values ​​include, for example, bot, spider, crawl, etc. An empty User-Agent value is also abnormal. These User-Agents (UAs) are the default UAs carried by some automated framework crawlers or the UAs of publicly available crawlers. If the first proportion is greater than a first threshold, it indicates that the terminal device is using an abnormal browser, meaning the access logs in this group were generated by access requests initiated by the terminal device through an abnormal browser. Therefore, the platform server will label the access logs in this group as negative samples. If the first proportion is less than or equal to the first threshold, it indicates that the terminal device is using a normal browser, and the platform server will label the access logs in this group as positive samples.

[0105] In another approach, the platform server determines the length of the User-Agent value in each access log and checks whether this length is greater than a first length or less than a second length. The first length is, for example, the maximum length of the User-Agent value, and the second length is, for example, the minimum length of the User-Agent value. If the length of the User-Agent value is greater than the first length, it indicates that the User-Agent value is too long; if the length of the User-Agent value is less than the second length, it indicates that the User-Agent value is too short; if the length of the User-Agent value is between the second length and the first length, it indicates that the length of the User-Agent value is normal.

[0106] Next, the platform server determines the number of excessively long and excessively short User-Agent values. Based on this number and the total number of access logs in the group, a second percentage is determined. If the second percentage is greater than a second threshold, it indicates that the terminal device is using an abnormal browser, meaning the access logs in this group were generated by access requests initiated by the terminal device through an abnormal browser. Therefore, the platform server will label the access logs in this group as negative samples. If the second percentage is less than or equal to the second threshold, it indicates that the terminal device is using a normal browser, and the platform server will label the access logs in this group as positive samples.

[0107] It should be noted that, although in the above embodiments, the platform server labels access logs based on a first quantity, a first proportion, or a second proportion, i.e., labels access logs based on a single dimension, this application embodiment is not limited to this. In other feasible implementations, the platform server may also label access logs based on at least two of the first quantity, the first proportion, and the second proportion, i.e., combine multi-dimensional labeling of access logs.

[0108] Using this approach, the platform server labels access logs based on the User-Agent value of each access log in the group, achieving the goal of automated and accurate labeling of access logs.

[0109] In the above embodiments, the platform server labels all access logs within a group as positive or negative samples based on the number of unique User-Agent values ​​and the proportion of access logs with abnormal User-Agent values. However, this embodiment is not limited. For example, when the constraint parameters include User-Agent values, the platform server can also determine whether the User-Agent value of each access log is a forged value. If the User-Agent value of the access log is forged, the access log is labeled as a negative sample; if the User-Agent value of the access log is not forged, the access log is labeled as a positive sample.

[0110] For example, due to the wide variety of browsers and the lack of clear, hard standards, they are easily forged, and it's easy to overlook whether a User-Agent value is forged. For instance, a User-Agent value might be Mozilla / 5.0 (Windows NT 6.2; WOW64) AppleWebKit / 537.36 (KHTML, like Gecko) Chrome / 27.0.1453.93 Safari / 537.36. Based on this User-Agent value, we can conclude that the browser on the terminal device is Chrome v27, i.e., version 27. However, the latest versions exceed 100. Considering the browser's release date, Chrome v27 was likely released around 2012, and it's now 2023. Clearly, the User-Agent value is forged. Different browsers have distinct User-Agent values ​​that exhibit strong characteristics; therefore, whether an access log is a positive or negative sample can be determined based on whether the User-Agent value is forged.

[0111] Using this approach, the platform server labels each access log entry based on whether the User-Agent value in each access log within a group is forged, thus achieving the goal of automatically and accurately labeling access logs.

[0112] Optionally, in the above embodiments, when the constraint parameters include the value of Referer, status code, attack type, or client unique identifier (CID), the platform server, in the process of labeling each access log in the group according to the constraint parameters, first determines the third proportion, fourth proportion, fifth proportion, or sixth proportion. Then, if the group meets at least one negative sample labeling condition, each access log in the group is labeled as a negative sample; if the group does not meet each negative sample labeling condition, each access log in the group is labeled as a positive sample. The third proportion is the proportion of access logs in the group containing the same Referer value (including null values); the fourth proportion is the proportion of access logs in the group containing an abnormal status code; the fifth proportion is the proportion of access logs in the group indicating the triggering of protection measures; the sixth proportion is the proportion of access logs in the group containing different CIDs; and the at least one negative sample labeling condition includes: the third proportion being greater than a third threshold, the fourth proportion being greater than a fourth threshold, the fifth proportion being greater than a fifth threshold, or the sixth proportion being greater than a sixth threshold. The platform server can flexibly label access logs according to the third, fourth, fifth, or sixth proportion.

[0113] For example, the platform server determines that access logs with empty referer values ​​in a group represent the third largest proportion of the total access logs. If this third proportion exceeds a third threshold, the access logs for that group are considered abnormal and are all marked as negative samples. If the third proportion is less than or equal to the third threshold, the access logs for that group are considered normal traffic access logs and are all marked as positive samples.

[0114] For example, the platform server determines the fourth percentage of access logs containing abnormal status codes within a group. If the fourth percentage exceeds a fourth threshold, the access logs for that group are considered abnormal and are all marked as negative samples. If the fourth percentage is less than or equal to the fourth threshold, the access logs for that group are considered normal traffic access logs and are all marked as positive samples.

[0115] For example, the platform server determines the fifth percentage of access logs that trigger protection measures within a group. If the fifth percentage exceeds the fifth threshold, the access logs for that group are considered abnormal and are all marked as negative samples. If the fifth percentage is less than or equal to the fifth threshold, the access logs for that group are considered normal traffic access logs and are all marked as positive samples.

[0116] Using this approach, the platform server labels access logs based on the Referer value, status code, attack type, or client unique identifier (CID) of each access log in the group, thereby achieving the goal of automated and accurate labeling of access logs.

[0117] It should be noted that, although in the above embodiments, when the platform server labels access logs based on the User-Agent value, Referer value, status code, attack type, or client unique identifier (CID), it does not distinguish between the business types of the groups, such as WEB business or API business. However, in actual implementation, different rules can be set for different business types. For example, when labeling access logs based on the User-Agent value, different first quantities, first proportions, and second proportions can be set for WEB business and API business, respectively.

[0118] For example, when the platform server marks access logs based on the Referer value, status code, attack type, or client unique identifier CID, different third, fourth, fifth, or sixth percentages can be set for WEB business and API business respectively.

[0119] Figure 3 This is a schematic diagram illustrating the sample annotation method provided in the embodiments of this application. Please refer to... Figure 3 After the platform server obtains the dataset to be labeled, it structures each access log in the dataset. The information recorded in each structured access log includes, but is not limited to, IP address, attack type, access path, access time, status code, User-Agent value, Referer value, CID, etc.

[0120] Next, the platform server divides the dataset to be labeled according to IP address and domain name, thus obtaining multiple groups, and grouping access logs with the same IP address and the same domain name into the same group.

[0121] Then, for each group, the platform server determines the business type of that group, such as web business or API business.

[0122] Subsequently, for each group, the platform server annotates each access log within that group according to all rules corresponding to the business type. These rules include, but are not limited to, the following: a first quantity greater than a preset quantity; a first percentage greater than a first threshold; a second percentage greater than a second threshold; a third percentage greater than a third threshold; a fourth percentage greater than a fourth threshold; a fifth percentage greater than a fifth threshold; or a sixth percentage greater than a sixth threshold.

[0123] Figure 3 In this context, rules 1 through N are, for example, rules corresponding to the first quantity, the first percentage, and the second percentage, and rules A through X are, for example, rules corresponding to the first quantity, the first percentage, and the second percentage, etc. It is understandable that the preset quantity, first threshold, and second threshold values ​​for web services may differ from those for API services.

[0124] Optionally, in the above embodiments, the platform server can also prioritize the various rules. Once a group meets a certain rule, all access logs within that group are marked as negative samples, and the judgment ends early, meaning that subsequent rules are no longer executed. The basis for marking the group as a negative sample is recorded. If none of the rules are met, the group is considered to be access logs generated by normal access, and the access logs within that group are marked as positive samples.

[0125] Optionally, in the above embodiments, after obtaining the labeled dataset, the platform server trains an initial model using the training dataset to obtain the crawler detection model and deploys it. Then, after receiving an access request from a client to access the target website, the platform server inputs the access request into the crawler detection model, causing the crawler detection model to output a detection result. The detection result indicates whether the access request was initiated by a crawler program. When the access request is initiated by a crawler program, the access request is processed.

[0126] For example, the platform server iteratively trains the initial model using training data until the model converges, thus obtaining a crawler detection model. This crawler detection model is then deployed on the platform server to detect crawler activity on access requests. If an access request is initiated by a crawler program, it is processed, such as by interception or monitoring. If an access request is a legitimate request, it is forwarded to the website server.

[0127] This approach utilizes a web crawler detection model to identify access requests, thereby achieving the goal of quickly and accurately detecting web crawler programs.

[0128] The following are embodiments of the apparatus described in this application, which can be used to execute the embodiments of the method described in this application. For details not disclosed in the apparatus embodiments of this application, please refer to the embodiments of the method described in this application.

[0129] Figure 4 This is a schematic diagram of a sample labeling device provided in an embodiment of this application. The sample labeling device 400 includes: an acquisition module 41, a processing module 42, and a labeling module 43.

[0130] The acquisition module 41 is used to acquire the dataset to be labeled, which contains multiple access logs to be labeled, and the access logs correspond one-to-one with the access requests initiated by the client to access the target website;

[0131] Processing module 42 is used to divide the dataset to be labeled according to the IP address and domain name of each access log to obtain multiple groups, and the IP address or domain name of the access logs in different groups are different.

[0132] The annotation module 43 is used to annotate each access log in the group according to the constraint parameters contained in each access log in the group for each of the plurality of groups, so as to obtain a training dataset for training the crawler detection model.

[0133] In one feasible implementation, the processing module 42 is further configured to determine the service type of each of the plurality of groups based on the constraint parameters contained in the access logs of each group;

[0134] The labeling module 43 is used to label each access log in the group according to the service type of the group.

[0135] In one feasible implementation, the processing module 42 is configured to, when the constraint parameter is a URL, extract a Uniform Resource Locator (URL) from each access log contained in the group to obtain multiple URLs for each of the multiple groups; determine a first type of URL and a second type of URL from the multiple URLs, wherein the first type of URL is a URL that loads static resources and the second type of URL is a URL that does not need to load static resources; when the proportion of the first type of URL in the multiple URLs is less than a first threshold, determine that the business type of the group is an API business; when the proportion of the first type of URL in the multiple URLs is greater than or equal to the first threshold, determine that the business type of the group is a WEB business.

[0136] In one feasible implementation, the processing module 42 is further configured to, when the business type of the group is a WEB business, determine an access sequence based on multiple URLs of the group, wherein the access sequence contains at least two URLs arranged in sequence; and determine the information entropy of the access sequence;

[0137] The annotation module 43 is used to annotate each access log in the group according to the information entropy.

[0138] In one feasible implementation, the annotation module 43 is used to extract the constraint parameters from each access log contained in the group. The constraint parameters include at least one of the following parameters: the value of User-Agent, the value of Referer, the status code, the attack type, or the client unique identifier CID. The attack type is used to indicate the protection measures triggered by the access request corresponding to the access log. The annotation module 43 is used to annotate each access log in the group according to the constraint parameters.

[0139] In one feasible implementation, the labeling module 43 is used to determine a first quantity, a first proportion, and a second proportion when the constraint parameters include User-Agent values. The first quantity is the number of User-Agent values ​​after deduplication of the group, the first proportion is the proportion of access logs in the group containing abnormal User-Agent values, and the second proportion is the proportion of access logs in the group containing User-Agent values ​​with a length greater than a first length or less than a second length. When the group meets at least one UA condition, each access log in the group is labeled as a negative sample. The at least one UA condition includes: the first quantity is greater than a preset quantity, the first proportion is greater than a first threshold, and the second proportion is greater than a second threshold.

[0140] In one feasible implementation, the labeling module 43 is used to determine whether the User-Agent value of the access log is a forged value for each access log in the group when the constraint parameter includes the value of User-Agent; and to label the access log as a negative sample when the User-Agent value of the access log is a forged value.

[0141] In one feasible implementation, the labeling module 43 is used to determine a third proportion, a fourth proportion, a fifth proportion, or a sixth proportion when the constraint parameters include the value of Referer, a status code, an attack type, or a client unique identifier (CID). The third proportion is the proportion of access logs in the group that contain the same Referer value, where the same Referer value includes null values. The fourth proportion is the proportion of access logs in the group that contain an abnormal status code. The fifth proportion is the proportion of access logs in the group that indicate the triggering of protection measures. The sixth proportion is the proportion of access logs in the group that contain different CIDs. When the group meets at least one negative sample labeling condition, each access log in the group is labeled as a negative sample. The at least one negative sample labeling condition includes: the third proportion being greater than a third threshold, the fourth proportion being greater than a fourth threshold, the fifth proportion being greater than a fifth threshold, or the sixth proportion being greater than a sixth threshold.

[0142] The sample labeling device provided in this application embodiment can execute the actions of the platform server in the above embodiment. Its implementation principle and technical effect are similar, and will not be described again here.

[0143] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device 500 is, for example, the platform server described above, and includes:

[0144] Processor 51 and memory 52;

[0145] The memory 52 stores computer instructions;

[0146] The processor 51 executes the computer instructions stored in the memory 52, causing the processor 51 to execute the sample annotation method implemented by the platform server as described above.

[0147] The specific implementation process of processor 51 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0148] Optionally, the electronic device 500 also includes a communication component 53. The processor 51, memory 52, and communication component 53 can be connected via a bus 54.

[0149] This application also provides a computer-readable storage medium storing computer instructions, which, when executed by a processor, are used to implement the sample annotation method implemented by the above-mentioned platform server.

[0150] This application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the sample annotation method implemented by the platform server described above.

[0151] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the application disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0152] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A sample labeling method, characterized by, include: Obtain the dataset to be labeled, which contains multiple access logs to be labeled, and the access logs correspond one-to-one with the access requests initiated by the client to access the target website; The dataset to be labeled is divided into multiple groups based on the IP address and domain name of each access log. The IP address or domain name of the access logs in different groups are different. For each of the plurality of groups, the business type of the group is determined as either a web business or an API business based on the URL in the constraint parameters contained in each access log of the group. For each of the multiple groups, the access logs in the group are labeled according to the business type of the group to obtain a training dataset for training the crawler detection model; For each of the plurality of groups, the access logs in the group are labeled according to the service type of the group, including: When the business type of the group is a web business, an access sequence is determined based on multiple URLs of the group, and the access sequence contains at least two URLs arranged in sequence; Determine the information entropy of the access sequence; Each access log in the group is labeled according to the information entropy; For each of the plurality of groups, the access logs in the group are labeled according to the service type of the group, including: The constraint parameters are extracted from each access log contained in the group. The constraint parameters include at least one of the following parameters: the value of User-Agent, the value of Referer, the status code, the attack type, or the client unique identifier CID. The attack type is used to indicate the protection measures triggered by the access request corresponding to the access log. Each access log in the group is labeled according to the constraint parameters and the service type of the group.

2. The method of claim 1, wherein, For each of the plurality of groups, determining whether the business type of the group is a web service or an API service based on the URL in the constraint parameters contained in each access log of the group includes: When the constraint parameter is a URL, for each of the multiple groups, a Uniform Resource Locator (URL) is extracted from each access log contained in the group to obtain multiple URLs; From the plurality of URLs, a first type of URL and a second type of URL are determined. The first type of URL is a URL that loads static resources, and the second type of URL is a URL that does not need to load static resources. When the proportion of the first type of URLs in a group of URLs is less than a preset threshold, the business type of the group is determined to be API business; When the proportion of the first type of URLs in a group of URLs is greater than or equal to a preset threshold, the business type of the group is determined to be WEB business.

3. The method of claim 1, wherein, The step of labeling each access log in the group according to the constraint parameters and the service type of the group includes: When the constraint parameter includes the value of User-Agent, a first quantity, a first proportion, and a second proportion are determined according to the business type of the group. The first quantity is the number of duplicate User-Agent values ​​in the group. The first proportion is the proportion of access logs in the group that contain abnormal User-Agent values. The second proportion is the proportion of access logs in the group that contain User-Agent values ​​with a length greater than the first length or less than the second length. When the group meets at least one UA condition, each access log in the group is marked as a negative sample. The at least one UA condition includes: the first quantity is greater than a preset quantity, the first proportion is greater than a first threshold, and the second proportion is greater than a second threshold.

4. The method of claim 1, wherein, The step of labeling each access log in the group according to the constraint parameters and the service type of the group includes: When the constraint parameter includes a User-Agent value, for each access log in the group, it is determined whether the User-Agent value of the access log is a forged value based on the business type of the group; When the User-Agent value of the access log is a forged value, the access log is marked as a negative sample.

5. The method of claim 1, wherein, The step of labeling each access log in the group according to the constraint parameters and the service type of the group includes: When the constraint parameters include the value of Referer, status code, attack type, or client unique identifier (CID), a third percentage, fourth percentage, fifth percentage, or sixth percentage is determined based on the business type of the group. The third percentage is the percentage of access logs in the group that contain the same Referer value, which includes null values. The fourth percentage is the percentage of access logs in the group that contain an abnormal status code. The fifth percentage is the percentage of access logs in the group that indicate the triggering of protection measures. The sixth percentage is the percentage of access logs in the group that contain different CIDs. When the group meets at least one negative sample labeling condition, each access log in the group is labeled as a negative sample. The at least one negative sample labeling condition includes: the third proportion is greater than the third threshold, the fourth proportion is greater than the fourth threshold, the fifth proportion is greater than the fifth threshold, or the sixth proportion is greater than the sixth threshold.

6. An electronic device comprising a processor, a memory, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the computer program, it causes the electronic device to implement the method as described in any one of claims 1 to 5.

7. A computer-readable storage medium having stored thereon a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 5.

Images