Anti-crawler method and device

Abstract

The invention discloses an anti-crawler method and device, relates to the technical field of anti-crawler, and aims to reduce the omission ratio and misjudgment rate of anti-crawler. According to the main technical scheme, the method comprises the steps of determining a target access parameter value in a real-time access log of a target user to a target object; determining a target time period according to a target access occurrence time point corresponding to the real-time access log; based on a target access log of which the access occurrence time point is located in a target time period in a database, determining access behavior characteristics and access parameter characteristics of the target user, the database being used for continuously collecting the access log of the target user to the target object; on the basis of the target access parameter value, the access behavior feature and the access parameter feature, crawler recognition is performed on the target user; and based on the crawler identification result, generating an anti-crawler result for indicating whether to execute an anti-crawler operation on the target user.

Description

Technical Field

[0001] The present application relates to the field of anti-crawler technology, and in particular to an anti-crawler method and device. Background Art

[0002] With the explosive growth of internet data, web crawlers have become an important tool for data collection. However, excessive or malicious use of web crawlers can lead to resource consumption and data leakage for service providers such as servers. This is why anti-crawler technology has emerged.

[0003] Existing anti-crawler technologies include the following two: The first is request header verification technology (such as User-Agent verification), which determines whether the accessing user is a crawler by detecting fields such as User-Agent in the request header of the access request. However, some crawlers can bypass verification by forging fields in the request header into fields corresponding to normal users (that is, real users, not crawlers), resulting in the inability to identify such crawlers. The second is IP restriction technology, which identifies crawlers by analyzing the IP address of the accessing user. However, some normal users may access the service provider through a shared IP or proxy server IP, and such normal users are often misjudged as crawlers.

[0004] Therefore, how to reduce the missed detection rate and false positive rate of anti-crawler has become an urgent problem that needs to be solved. Summary of the Invention

[0005] This application proposes an anti-crawler method and device, the main purpose of which is to reduce the missed detection rate and false positive rate of anti-crawler.

[0006] In order to achieve the above objectives, this application mainly provides the following technical solutions:

[0007] In the first aspect, the present application provides an anti-crawler method, which may include: determining the target access parameter value in the real-time access log of the target user to the target object; determining the target time period based on the target access occurrence time point corresponding to the real-time access log; determining the access behavior characteristics and access parameter characteristics of the target user based on the target access log in the database whose access occurrence time point is within the target time period, the database is used to continuously collect the access log of the target user to the target object; performing crawler identification on the target user based on the target access parameter value, the access behavior characteristics and the access parameter characteristics; and generating an anti-crawler result based on the crawler identification result to indicate whether to perform an anti-crawler operation on the target user.

[0008] In a second aspect, the present application provides an anti-crawler device, which may include:

[0009] A first determining module is used to determine a target access parameter value in a real-time access log of a target user to a target object;

[0010] A second determining module is configured to determine a target time period according to a target access occurrence time point corresponding to the real-time access log;

[0011] a third determining module, configured to determine the access behavior characteristics and access parameter characteristics of the target user based on a target access log in a database where the access time point is within the target time period, wherein the database is configured to continuously collect the target user's access log to the target object;

[0012] an identification module, configured to perform crawler identification on the target user based on the target access parameter value, the access behavior characteristics, and the access parameter characteristics;

[0013] A generation module is used to generate an anti-crawler result based on the crawler identification result, which is used to indicate whether to perform an anti-crawler operation on the target user.

[0014] In a third aspect, the present application provides a computer-readable storage medium, which includes a stored program, wherein when the program is running, the device where the storage medium is located is controlled to execute the anti-crawler method described in the first aspect.

[0015] In a fourth aspect, the present application provides an electronic device comprising: a memory for storing a program; and a processor coupled to the memory for running the program to execute the anti-crawler method described in the first aspect.

[0016] In a fifth aspect, the present application provides a computer program product, which includes: a computer program / computer executable instructions, which, when executed by a processor, implements the anti-crawler method described in the first aspect.

[0017] The anti-crawl method and apparatus provided herein, after identifying a target user, determine the target access parameter values in the target user's real-time access logs to the target object. A target time period is then determined based on the target access time points corresponding to the real-time access logs. The target user's access behavior characteristics and access parameter characteristics are then determined based on the target access logs in the database whose access time points fall within the target time period. The target user is then identified as a crawler based on the target access parameter values, access behavior characteristics, and access parameter characteristics. Finally, based on the crawler identification results, an anti-crawl result is generated, indicating whether to perform an anti-crawl operation on the target user. Thus, the solution provided by this embodiment can achieve at least the following three effects: First, when performing an anti-crawl operation, the solution provided by this embodiment determines instantaneous features, such as access parameters, in the user's real-time access logs at discrete time points, as well as long-term behavioral features, such as access behavior characteristics and access parameter characteristics, over a time series, i.e., the target time period. Finally, based on these two dimensions of features, the user is identified as a crawler and an anti-crawl result is generated for the target user based on the crawler identification results. In this way, since the anti-crawler operation is performed by integrating the features of the two dimensions, the missed detection rate and false positive rate of the anti-crawler can be reduced. Secondly, the solution provided by this embodiment does not require the intervention of the target user who accesses the target object when performing the anti-crawler operation. The entire anti-crawler operation is imperceptible to the target user, thus reducing the negative impact of the anti-crawler operation on the user experience. Thirdly, when performing the anti-crawler operation, the solution provided by this embodiment is based on the target user's access log to the target object. There is no need to encrypt the business content in the target object for user access, nor is there any need to obfuscate the code of the target object. Therefore, the anti-crawler operation provided by this embodiment will not affect the business content and code of the target object itself.

[0018] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0020] Figure 1 A flowchart of an anti-crawler method provided by an embodiment of the present application is shown;

[0021] Figure 2 A flowchart of an anti-crawler method provided by another embodiment of the present application is shown;

[0022] Figure 3 A schematic structural diagram of an anti-crawler device provided in one embodiment of the present application is shown;

[0023] Figure 4 A structural schematic diagram of an anti-crawler device provided in another embodiment of the present application is shown. DETAILED DESCRIPTION

[0024] Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0025] Excessive or malicious use of crawlers can cause the following problems for service providers, such as servers: First, valuable data (such as product prices and user reviews) from service providers (such as product prices and user reviews) can be captured in bulk by crawlers, leading to data theft and misuse by competitors. Second, the high frequency of crawler requests consumes significant bandwidth and computing resources, causing delays in normal user access and even service crashes. Third, crawlers capture sensitive data (such as user information), leading to data leaks. Therefore, anti-crawler technology has emerged to protect service providers' data security and maintain stable operations.

[0026] Existing anti-scraping technologies face the dual challenges of missed detection (e.g., failing to identify malicious bots) and false positives (e.g., incorrectly blocking legitimate users). This challenge is exacerbated by the increasing sophistication of bot behavior and advancements in camouflage techniques. Therefore, reducing the missed detection and false positive rates of anti-scraping technologies has become a pressing issue.

[0027] Research has shown that crawler behavior is not only reflected in independent access logs at discrete points in time, but also requires comprehensive identification through analysis of access logs over time series. Therefore, anti-crawler operations can determine a user's instantaneous characteristics based on independent access logs at discrete points in time, and their long-term behavioral characteristics based on access logs over time series. Combining these instantaneous and long-term behavioral characteristics allows for a comprehensive assessment of whether a user is a crawler, reducing both missed detections and false positives.

[0028] Based on the above findings, this embodiment specifically provides an anti-crawler technical solution, which specifically includes: determining the target access parameter value (i.e., instantaneous feature) in the real-time access log of the target user to the target object. Determine the target time period based on the target access occurrence time point corresponding to the real-time access log. Based on the target access log in the database (the database is used to continuously collect the access log of the target user to the target object) whose access occurrence time point is within the target time period, determine the access behavior characteristics and access parameter characteristics (i.e., long-term behavior characteristics) of the target user. Based on the target access parameter value, access behavior characteristics, and access parameter characteristics, perform crawler identification on the target user. Based on the crawler identification result, generate an anti-crawler result for indicating whether to perform anti-crawler operations on the target user.

[0029] Based on the above anti-crawler technical solution, this embodiment specifically provides an anti-crawler method and device. The anti-crawler method and device provided in this embodiment are specifically described below.

[0030] The present application embodiment provides an anti-crawler method, such as Figure 1 As shown, the anti-crawler method provided in this embodiment may include at least the following steps 101 to 104:

[0031] 101. Determine a target access parameter value in a real-time access log of a target user to a target object.

[0032] The target object is a service provider that is accessible to users, provides services to users, and requires crawler access restrictions. The type of target object can be flexibly selected based on business needs and is not limited in this embodiment. Exemplary service providers include, but are not limited to, servers and cloud platforms. The target user is the user who accesses the target object. This user can be any user of the target object, or a user whose identity as a legitimate user of the target object is unclear, and is not limited in this embodiment.

[0033] After determining the target object, the target user's real-time access logs to the target object are continuously collected to obtain the user's instantaneous features based on the real-time access logs at discrete time points. This allows for identification of the target user as a crawler based on these instantaneous features. Considering that crawler behavior is often reflected in access parameter values, target access parameter values are determined from the target user's real-time access logs to the target object, and used as instantaneous features.

[0034] The specific process of determining the target access parameter value in the real-time access log of the target user to the target object may include: selecting the target access parameter corresponding to the business type of the target object from the preset correspondence between the business type and the target access parameter; and determining the access parameter value of the target access parameter in the real-time access log of the target user to the target object.

[0035] The access parameter values associated with the target object's business type reflect the crawler's behavior. Therefore, to more quickly extract target access parameter values suitable for crawler identification, a pre-defined correspondence between business types and target access parameters is established. Within this correspondence, the target access parameter values support crawler identification of users accessing objects of the corresponding business type. In other words, the target access parameter values are crucial data for distinguishing crawlers from legitimate users.

[0036] Furthermore, considering the parameter values of the request parameters involved in the target user's access to the target object and the parameter values of the target object's response parameters to the target user's access, both can be used to distinguish between crawlers and normal users. Based on this, the determined target access parameter values may include, but are not limited to, the parameter values of at least one of the following parameter types: the request parameters involved in the target user's access to the target object and the target object's response parameters to the target user's access.

[0037] 102. Determine a target time period based on the target access occurrence time point corresponding to the real-time access log.

[0038] After determining the target user's target access parameter values from their real-time access logs to the target object, which are instantaneous features, we need to determine the target user's long-term behavioral characteristics based on their access logs over time. This combined instantaneous and long-term behavioral features allows for more accurate identification of the target user as a crawler. The first step in identifying the target user's long-term behavioral characteristics over time is to determine the target time period corresponding to the time series, thereby filtering out access logs for determining long-term behavioral characteristics.

[0039] The specific process of determining the target time period according to the target access occurrence time point corresponding to the real-time access log may include the following steps 102A to 102B.

[0040] 102A. Determine a start time point and an end time point based on the target access occurrence time point.

[0041] The target access occurrence time point refers to the precise timestamp when the target user initiates access to the target object. It is recorded in the real-time access log and is important data for determining the start and end time points corresponding to the target time period.

[0042] The relationship between the start time point and the target access occurrence time point is: the start time point is before the target access occurrence time point. For example, the target access occurrence time point is T and the start time point is Ta, where Ta indicates that the start time point is before the target access occurrence time point and is separated from the target access occurrence time point by a time interval of a.

[0043] The relationship between the termination time point and the target access occurrence time point may include, but is not limited to, any of the following: The first is that the termination time point is before the target access occurrence time point and after the start time point. Exemplarily, the target access occurrence time point is T, the start time point is Ta, and the termination time point is Tb, b is less than a, and Tb indicates that the termination time point is before the target access occurrence time point and after the start time point, and the interval between the termination time point and the target access occurrence time point is b. The second is that the termination time point is after the target access occurrence time point. Exemplarily, the target access occurrence time point is T, the start time point is Ta, and the termination time point is T+c, and T+c indicates that the termination time point is after the target access occurrence time point, and the interval between the termination time point and the target access occurrence time point is c. The third is that the termination time point is the target access occurrence time point. Exemplarily, the target access occurrence time point is T, the start time point is Ta, and the termination time point is T.

[0044] 102B. Determine the time period from the start time point to the end time point as the target time period.

[0045] After determining the starting time point and the ending time point, the front and rear boundary values of the target time period are obtained. Therefore, the time period from the starting time point to the ending time point is determined as the target time period in order to determine the long-term behavioral characteristics of the target user in the time series of the target time period.

[0046] 103. Based on the target access log in the database where the access time point is within the target time period, the access behavior characteristics and access parameter characteristics of the target user are determined. The database is used to continuously collect the target user's access log to the target object.

[0047] In some embodiments, a database continuously collects target users' access logs to target objects, facilitating the extraction of access logs for use in determining long-term behavioral characteristics. Specifically, whenever a target user initiates an access to a target object, the corresponding access log is collected in the database. The type of database can be flexibly selected based on business needs and is not limited in this embodiment.

[0048] In some embodiments, after determining the target time period, a target access log in the database is identified whose access occurred within the target time period, and the target user's long-term behavioral characteristics over the time series of the target time period are determined based on the target access log within the target time period. This allows the target user to be identified as a crawler based on such long-term behavioral characteristics that support identifying whether the target user is a crawler. Considering that crawler behavior is typically reflected in access behavior and access parameters, the target user's access behavior characteristics and access parameter characteristics are used as long-term behavioral characteristics.

[0049] In some embodiments, based on a target access log in a database whose access occurs within a target time period, the specific process of determining the target user's access behavior characteristics and access parameter characteristics can be as follows: inputting the target access log into a first analysis model, and extracting the target user's access behavior characteristics in the time series of the target time period based on the target access log analysis by the first analysis model; inputting the target access log into a second analysis model, and extracting the target user's access parameter characteristics in the time series of the target time period based on the target access log analysis by the second analysis model. The first analysis model is obtained by training multiple sets of data and is a model for extracting access behavior characteristics from access log analysis, each set of data includes access behavior characteristics and access logs corresponding to the access behavior characteristics, and these access logs are generated within a first preset time period. The second analysis model is obtained by training multiple sets of data and is a model for extracting access parameter characteristics from access log analysis, each set of data includes access parameter characteristics and access logs corresponding to the access parameter characteristics, and these access logs are generated within a second preset time period.

[0050] Access behavior features are used to reflect the target user's access behavior to the target object over a target time period. Access behavior features may include, but are not limited to, at least one of the following: user access periodicity, user identification features, and user access frequency. User access periodicity indicates at least one of the following: the number of time units (e.g., days) the target user accessed within the target time period; the frequency of the target user's visits within the target time period; the average interval between visits within the target time period; the change in the number of visits per time unit by the target user within the target time period; and the change in the time period during which the target user visited each time unit within the target time period. User identification features indicate at least one of the following: the target user's fingerprint features, the target user's UA fingerprint, and the target user's IP address. User access frequency indicates at least one of the following: the number of visits per time unit by the target user within the target time period; the total number of visits per time segment (a time segment consists of multiple consecutive time units) by the target user within the target time period; the average number of visits per time unit by the target user within the target time period; the average number of visits per time segment by the target user within the target time period; the total number of visits by the target user within the target time period; and the target user's most frequently visited time segments within the target time period.

[0051] Access parameter features are used to reflect the personalized characteristics of the target user's access to the target object during the target time period. They can reflect the target user's behavioral preferences and access intentions. Access parameter features may include, but are not limited to, at least one of the following: the access parameter used by the target user when accessing the target object, the access parameter type, the access parameter value, and the attributes of the access parameter value.

[0052] 104. Based on the target access parameter value, access behavior characteristics and access parameter characteristics, the target user is identified as a crawler.

[0053] After determining the target access parameter values in the target user's real-time access log to the target object, which are instantaneous features, and determining the target user's access behavior features and access parameter features, which are long-term behavioral features, it is necessary to combine the instantaneous features and long-term behavioral features to identify the target user as a crawler, so as to more accurately determine whether the target user is a crawler.

[0054] The specific implementation method of performing crawler identification on the target user based on the target access parameter value, access behavior characteristics and access parameter characteristics may include at least the following method A1 and method A2.

[0055] Method A1, based on the target access parameter value, access behavior characteristics and access parameter characteristics, the specific process of performing crawler identification on the target user may include the following steps 104A to 104C.

[0056] 104A. Perform crawler identification on the target user based on the target access parameter value to obtain a first crawler identification result.

[0057] Performing crawler identification on the target user based on the target access parameter value is essentially performing crawler identification on the target user based on the instantaneous feature dimension. Therefore, the first crawler identification result indicates the likelihood that the target user is a crawler based on the instantaneous feature dimension. The specific process of performing crawler identification on the target user based on the target access parameter value and obtaining the first crawler identification result may include the following steps 104A1 to 104A3.

[0058] 104A1. Obtain the parameter threshold and target parameter value attributes corresponding to each target access parameter value at the time point when the target access occurs. The parameter threshold and target parameter value attributes are obtained based on the access log of the target object by normal users and are dynamically updated over time.

[0059] A correspondence between access parameters, parameter thresholds, and target parameter value attributes is set up to facilitate the rapid determination of the parameter thresholds and target parameter value attributes corresponding to each target access parameter value at the time point when the target access occurs. It should be noted that, taking into account the passage of time or changes in user needs, the parameter thresholds and target parameter value attributes corresponding to normal users will change. Therefore, in order to ensure the accuracy of anti-crawler, the parameter thresholds and target parameter value attributes in the aforementioned correspondence are dynamically updated over time. Exemplarily, the access logs of normal users to the target object in the current period are periodically obtained, and the latest parameter thresholds and target parameter value attributes corresponding to each access parameter are determined based on the access logs, and the aforementioned correspondence is updated based on the latest parameter thresholds and target parameter value attributes. It should be noted that the parameter threshold can be determined based on any one of the maximum value, minimum value, median, mode, and variance of the access parameter values of the corresponding access parameters in the access logs of normal users to the target object in the current period, and this embodiment does not limit this.

[0060] The specific process of obtaining the parameter threshold and target parameter value attributes corresponding to each target access parameter value at the time point when the target access occurs is to perform the following steps for each target access parameter value: query the aforementioned correspondence, determine the target access parameter corresponding to each target access parameter value, and determine the parameter threshold and target parameter value attributes corresponding to the target access parameter in the correspondence as the parameter threshold and target parameter value attributes corresponding to the target access parameter value.

[0061] 104A2. For each target access parameter value, respectively perform the following steps: compare the target access parameter value with its corresponding parameter threshold, and compare the parameter value attribute of the target access parameter value with its corresponding target parameter value attribute to obtain a comparison result corresponding to the target access parameter value.

[0062] In this embodiment, step 104A2 can be further refined to perform the following steps for each target access parameter value: detecting whether the target access parameter value is greater than its corresponding parameter threshold; if so, determining that the comparison result with the parameter threshold is that the target access parameter value is greater than the parameter threshold, and providing a corresponding first matching score based on the difference between the target access parameter value and the parameter threshold; if not, determining that the comparison result with the parameter threshold is that the target access parameter value is not greater than the parameter threshold, and providing a corresponding second matching score. Determining the similarity between the parameter value attribute of the target access parameter value and its corresponding target parameter value attribute, and determining a third matching score that matches the similarity as the comparison result with the target access parameter value attribute. Based on the first matching score, the second matching score, and the third matching score, a comparison result of the target access parameter value is obtained.

[0063] 104A3. Obtain a first crawler identification result based on the comparison result corresponding to each target access parameter value.

[0064] The comparison results for the target access parameter values indicate a corresponding score, which reflects the probability that the target user is a crawler, as assessed by the corresponding target access parameter value. Based on this, the comparison results for each target access parameter value are aggregated to produce a first crawler identification result. This first crawler identification result indicates the probability that the target user is a crawler, as assessed by the instantaneous feature dimension.

[0065] 104B. Perform crawler identification on the target user based on the access behavior characteristics and the access parameter characteristics to obtain a second crawler identification result.

[0066] Identifying a target user as a crawler based on access behavior and access parameter characteristics is essentially identifying the target user as a crawler based on long-term behavioral characteristics. Therefore, the second crawler identification result indicates the probability that the target user is a crawler based on long-term behavioral characteristics. Methods for identifying a target user as a crawler based on access behavior and access parameter characteristics to obtain the second crawler identification result may include at least the following methods: B1, B2, and B3.

[0067] Method B1 determines the normal access behavior characteristics and normal access parameter characteristics corresponding to normal users of the target object, and obtains the second crawler identification result based on the similarity between the access behavior characteristics of the target object and the normal access behavior characteristics, as well as the similarity between the access parameter characteristics of the target object and the normal access parameter characteristics.

[0068] In order to facilitate crawler identification of target users based on access behavior characteristics and access parameter characteristics, normal access behavior characteristics and normal access parameter characteristics corresponding to normal users are set in advance based on the historical normal users' access logs to the target object. In this way, when it is necessary to perform crawler identification of target users based on access behavior characteristics and access parameter characteristics, the normal access behavior characteristics and normal access parameter characteristics corresponding to normal users can be obtained for use. Furthermore, considering that the access behavior characteristics and access parameter characteristics corresponding to normal users will change with the passage of time or changes in user needs, in order to ensure the accuracy of anti-crawler, the access behavior characteristics and access parameter characteristics corresponding to normal users can be periodically updated, that is, the access behavior characteristics and access parameter characteristics corresponding to normal users are dynamically updated over time.

[0069] After determining the normal access behavior characteristics and normal access parameter characteristics corresponding to a normal user of the target object, a score corresponding to the similarity between the target object's access behavior characteristics and the normal access behavior characteristics is first determined, as well as a score corresponding to the similarity between the target object's access parameter characteristics and the normal access parameter characteristics. Then, the confidence level corresponding to the access behavior characteristics and the access parameter characteristics is obtained. Based on the confidence level, a weighted calculation is performed on the scores corresponding to the target user's access behavior characteristics and access parameter characteristics, and the result of the weighted calculation is determined as the second crawler identification result.

[0070] Method B2, based on the normal access behavior characteristics and normal access parameter characteristics corresponding to normal users of the target object, as well as the access behavior characteristics and access parameter characteristics of the target user, clusters the target user and normal users, and obtains the second crawler recognition result based on the clustering processing result.

[0071] Using a preset clustering algorithm, the target user and normal users are clustered based on the normal access behavior and parameter characteristics corresponding to normal users of the target object, as well as the target user's access behavior and parameter characteristics. If, after clustering, the target object and normal users are assigned to the same cluster, it indicates that the target user is likely a normal user. A second crawler identification result is obtained based on the score reflecting this fact. If, after clustering, the target object and normal users are not assigned to the same cluster, it indicates that the target user is likely a crawler. A second crawler identification result is obtained based on the score reflecting this fact.

[0072] Method B3 determines the target user's user identification features in the real-time access log. If the target user's user identification features indicate that the target user is a normal user, the degree of match between the target user's access behavior features and access parameter features and the target user's user identification features is determined. If the degree of match does not reach a matching threshold, a second crawler identification result is obtained, indicating that the target user is a crawler. If the degree of match reaches the matching threshold, a second crawler identification result is obtained, indicating that the target user is a non-crawler.

[0073] Specifically, the user identification features of the target user are matched with the user identification features corresponding to known normal users; if the user identification features of the normal user are successfully matched, it is determined that the user identification features of the target user indicate that the target user is a normal user; if the user identification features of the normal user are not successfully matched, it is determined that the user identification features of the target user indicate that the target user is a crawler.

[0074] Specifically, determining the degree of match between the target user's access behavior characteristics and access parameter characteristics and the target user's user identification characteristics may include: inputting the target user's access behavior characteristics and access parameter characteristics, as well as the target user's user identification characteristics, into a preset designated model, and having the preset designated model output the degree of match. The preset designated model is trained using multiple sets of data and is used to output the degree of match between the access behavior characteristics and access parameter characteristics and the user identification characteristics. Each set of data includes the degree of match and the corresponding access behavior characteristics, access parameter characteristics, and user identification characteristics.

[0075] Specifically, after obtaining the matching degree, the matching degree is compared with the matching threshold. If the matching degree does not reach the matching threshold, it indicates that the user identification feature of the target user is likely to be forged by a crawler, and thus a second crawler identification result is obtained, indicating that the target user is a crawler. If the matching degree reaches the matching threshold, it indicates that the user identification feature of the target user is likely to be a genuine user identification feature of a normal user, and thus a second crawler identification result is obtained, indicating that the target user is not a crawler.

[0076] At least one of the above methods B1, B2, and B3 can be selected based on business needs. When two or all three are selected, the second crawler identification results of the selected methods are cross-checked. It should be noted that both methods B1 and B2 perform outlier analysis on the target user based on their access behavior characteristics and access parameter characteristics to identify whether the target user is a crawler.

[0077] 104C. Determine a crawler recognition result for the target user based on the first crawler recognition result and the second crawler recognition result.

[0078] Based on the first crawler identification result and the second crawler identification result, the specific process of determining the crawler identification result for the target user may include the following steps: based on the first crawler identification result and the second crawler identification result and their respective corresponding confidence levels, weighted calculation is performed on the probability indicated by the first crawler identification result and the probability indicated by the second crawler identification result; if the weighted calculation result is greater than the probability threshold, a crawler identification result indicating that the target user is a crawler is obtained; if the weighted calculation result is not greater than the probability threshold, a crawler identification result indicating that the target user is a non-crawler is obtained.

[0079] The confidence level of the first crawler recognition result reflects the credibility of the crawler recognition result derived from the instantaneous feature dimension. The confidence level of the second crawler recognition result reflects the credibility of the crawler recognition result derived from long-term behavioral features. In this way, based on the confidence level, the probability indicated by the first crawler recognition result and the probability indicated by the second crawler recognition result are weighted and calculated, resulting in a more accurate crawler recognition result.

[0080] Method A2, the specific process of identifying a target user as a crawler based on a target access parameter value, access behavior characteristics, and access parameter characteristics, may include the following steps: inputting the target access parameter value, access behavior characteristics, and access parameter characteristics into a preset model; and determining a crawler identification result for the target user based on the target access parameter value, access behavior characteristics, and access parameter characteristics using the preset model. The preset model is trained using multiple sets of data and is used to identify whether a user is a crawler based on the target access parameter value, access behavior characteristics, and access parameter characteristics. Each set of data includes the target access parameter value, access behavior characteristics, access parameter characteristics, and the crawler identification result corresponding to the target access parameter value, access behavior characteristics, and access parameter characteristics.

[0081] At least one of the above methods A1 and A2 can be selected for use based on business needs. When both methods are selected, the crawler recognition results of the two methods are mutually verified.

[0082] 105. Based on the crawler identification result, generate an anti-crawler result for indicating whether to perform an anti-crawler operation on the target user.

[0083] The crawler identification result corresponding to the target user is used to indicate whether the target user is a crawler. If the anti-crawler result is generated only based on the crawler identification result indicating whether the target user is a crawler, the implementation process of generating the anti-crawler result for indicating whether to perform anti-crawler operations on the target user based on the crawler identification result can include the following two methods:

[0084] One is that if the crawler identification result indicates that the target user is not a crawler, it means that the target user is a normal user. At this time, there is no need to intervene in the target user's access to the target object, and the target user's access to the target object does not need to be interfered with. Therefore, an anti-crawler result is generated to indicate that anti-crawler operations should not be performed on the target user.

[0085] Another approach is to generate an anti-crawler result instructing the target user to perform an anti-crawler operation on the target user if the crawler identification result indicates that the target user is a crawler. In this implementation, the specific behavior of the crawler does not need to be considered. As long as the crawler identification result indicates that the target user is a crawler, an anti-crawler result instructing the target user to perform an anti-crawler operation on the target user is generated, thereby performing the anti-crawler operation on the crawler.

[0086] Furthermore, if other factors need to be considered in addition to the crawler identification result, the implementation process of generating an anti-crawler result for indicating whether to perform an anti-crawler operation on the target user based on the crawler identification result may include the following methods C1 to C3.

[0087] Method C1, if the crawler identification result indicates that the target user is a crawler, verification information is sent to the target user, and the verification information is used to instruct the target user to perform the target action, and the target action is an action that the crawler is unable to perform; if the target information triggered by the execution of the target action is received, an anti-crawler result is generated to indicate that the anti-crawler operation is not performed on the target user; if the target information triggered by the execution of the target action is not received, an anti-crawler result is generated to indicate that the anti-crawler operation is performed on the target user.

[0088] To further reduce the possibility of misidentifying legitimate users as crawlers, this embodiment sends verification information to the target user if the crawler identification result indicates that the target user is a crawler. The specific form of the verification information can be determined based on business needs and is not limited in this embodiment. For example, the verification information includes an image and a prompt to click a specified area in the image.

[0089] If the target information triggered by the execution of the target action is received, it means that the target user is a normal user and has performed the target action. The crawler identification result indicating that the target user is a crawler is likely a misjudgment, so an anti-crawler result is generated to indicate that the anti-crawler operation should not be performed on the target user.

[0090] If the target information triggered by the execution of the target action is not received, it means that the target user has not performed the target action, and it is very likely that the target user is a crawler. The crawler identification result indicates that there is no misjudgment that the target user is a crawler, so an anti-crawler result is generated to indicate that an anti-crawler operation is performed on the target user.

[0091] Method C2, if the crawler identification result indicates that the target user is a crawler, then detect whether the response data of the target object to the target user's access within the target time period carries target sensitive data, and the amount of target sensitive data carried is greater than the target threshold; if so, generate an anti-crawler result for indicating that an anti-crawler operation is performed on the target user; if not, generate an anti-crawler result for indicating that an anti-crawler operation is not performed on the target user.

[0092] If the crawler identification result indicates that the target user is a crawler, it is necessary to detect whether the response data of the target object to the target user's access within the target time period carries the target sensitive data, and the amount of target sensitive data carried is greater than the target threshold, so as to determine whether it is necessary to perform anti-crawler operations on the target user.

[0093] If it is detected that the response data of the target object to the target user's access within the target time period carries the target sensitive data, and the amount of the target sensitive data carried is greater than the target threshold, it means that the target user has caused a large amount of sensitive data leakage, and it is necessary to perform anti-crawler operations on the target user. Therefore, an anti-crawler result is generated to indicate that the anti-crawler operation is to be performed on the target user.

[0094] If it is detected that the response data of the target object to the target user's access within the target time period does not carry the target sensitive data, or if it is detected that the response data of the target object to the target user's access within the target time period carries the target sensitive data, and the amount of target sensitive data carried is not greater than the target threshold, it means that the target user has not caused a large amount of sensitive data leakage, and the anti-crawler operation can be temporarily not performed on the target user, so an anti-crawler result is generated to indicate that the anti-crawler operation will not be performed on the target user.

[0095] Method C3, if the crawler identification result indicates that the target user is a crawler, then detect whether the target user's access to the target object within the target time period affects the performance of the target object; if so, generate an anti-crawler result for instructing to perform anti-crawler operations on the target user; if not, generate an anti-crawler result for instructing not to perform anti-crawler operations on the target user.

[0096] If the crawler identification result indicates that the target user is a crawler, it is necessary to detect whether the target object's access to the target user during the target time period affects the performance of the target object, so as to determine whether it is necessary to perform anti-crawler operations on the target user.

[0097] If it is detected that the target object's access to the target user within the target time period affects the performance of the target object, it means that the target user's access has affected the normal operation or service of the target object, and it is necessary to perform anti-crawler operations on the target user. Therefore, an anti-crawler result is generated to indicate that anti-crawler operations are performed on the target user.

[0098] If it is detected whether the target user's access to the target object within the target time period affects the performance of the target object, it means that the target user has not affected the normal operation or service of the target object, and anti-crawler operations can be temporarily not performed on the target user, thereby generating an anti-crawler result indicating that anti-crawler operations will not be performed on the target user.

[0099] The above methods C1 to C3 can be selected for use by one method or at least two methods can be selected for use in combination based on business needs, which is not limited in this embodiment.

[0100] The anti-crawl method provided in an embodiment of the present application, after identifying a target user, determines the target access parameter values in the target user's real-time access logs to the target object. A target time period is then determined based on the target access time points corresponding to the real-time access logs. The target user's access behavior characteristics and access parameter characteristics are then determined based on the target access logs in the database whose access time points fall within the target time period. The target user is then identified as a crawler based on the target access parameter values, access behavior characteristics, and access parameter characteristics. Finally, based on the crawler identification results, an anti-crawl result is generated, indicating whether to perform an anti-crawl operation on the target user. Thus, the solution provided in this embodiment can achieve at least the following three effects: First, when performing an anti-crawl operation, the solution provided in this embodiment determines instantaneous features, such as access parameters, in the user's real-time access logs at discrete time points, as well as long-term behavioral features, such as access behavior characteristics and access parameter characteristics, over a time series, i.e., the target time period. Finally, the solution combines these two dimensions of features to identify the user as a crawler, and generates an anti-crawl result for the target user based on the crawler identification results. In this way, since the anti-crawler operation is performed by integrating the features of the two dimensions, the missed detection rate and false positive rate of the anti-crawler can be reduced. Secondly, the solution provided by this embodiment does not require the intervention of the target user who accesses the target object when performing the anti-crawler operation. The entire anti-crawler operation is imperceptible to the target user, thus reducing the negative impact of the anti-crawler operation on the user experience. Thirdly, when performing the anti-crawler operation, the solution provided by this embodiment is based on the target user's access log to the target object. There is no need to encrypt the business content in the target object for user access, nor is there any need to obfuscate the code of the target object. Therefore, the anti-crawler operation provided by this embodiment will not affect the business content and code of the target object itself.

[0101] Furthermore, an embodiment of the present application provides an anti-crawler method, such as Figure 2 As shown, the anti-crawler method provided in this embodiment may further include the following steps 201 to 212.

[0102] 201. Determine a target user identification feature of the target user in a real-time access log of the target user to the target object.

[0103] The target user identification feature is a feature that can quickly identify whether the target user is a crawler, which may include but is not limited to IP address, UA information, etc.

[0104] 202. Determine whether the target user identification feature matches the user identification feature of the crawler indicated by the blacklist; if so, execute step 203; if not, execute step 204.

[0105] The blacklist is used to indicate user identification features known to be possessed by crawlers. If the target user identification feature is determined to be included in the blacklist, or if the similarity with any user identification feature in the blacklist reaches a threshold, then the target user identification feature is determined to match the blacklist, indicating that the target user is likely a crawler. Therefore, step 203 is executed to generate a crawler identification result indicating that the target user is a crawler.

[0106] 203 . Generate a crawler identification result indicating that the target user is a crawler, and execute step 208 .

[0107] 204. Determine a target access parameter value in a real-time access log of a target user to a target object.

[0108] 205. Determine a target time period based on the target access occurrence time point corresponding to the real-time access log.

[0109] 206. Based on the target access log in the database where the access time point is within the target time period, determine the access behavior characteristics and access parameter characteristics of the target user. The database is used to continuously collect the target user's access log to the target object.

[0110] 207. Based on the target access parameter value, access behavior characteristics, and access parameter characteristics, perform crawler identification on the target user.

[0111] 208. Based on the crawler identification result, generate an anti-crawler result for indicating whether to perform an anti-crawler operation on the target user, and execute steps 209 and 212.

[0112] 209. If the anti-crawler result indicates that an anti-crawler operation is to be performed on the target user, a first user feature of the target user and a second user feature of the accessing user are determined based on their respective access logs in the database.

[0113] Considering that an attacker may use multiple accounts as access users to steal data information, if a target user is identified as a crawler, it is necessary to identify the associated users so that anti-crawler operations can be performed on the associated users to prevent the associated users from continuing to access the target object.

[0114] The database is also used to continuously collect access logs of users other than the target user who access the target object. Based on this, in order to find the target user's associated users, a first user characteristic of the target user is determined based on the target user's access log in the database, and a second user characteristic of the access user is determined based on the access log of the access user in the database. The first user characteristic and the second user characteristic encompass at least one of the following categories of characteristics: access behavior characteristics, access parameter characteristics, user identification characteristics, and access parameter values. Detailed explanations of access behavior characteristics, access parameter characteristics, user identification characteristics, and access parameter values can be found in the above steps and will not be repeated here.

[0115] 210. Based on the first user feature of the target user and the second user feature of the visiting user, determine whether there is an associated user of the target user among the visiting users. If so, execute step 211; if not, end the current process.

[0116] The degree of feature matching between the first user feature of the target user and the second user feature of the visiting user is the key to identifying the target user's associated users. Based on this, based on the first user feature of the target user and the second user feature of the visiting user, the following two methods can be used to determine whether there are any associated users of the target user among the visiting users.

[0117] One method is to determine the similarity between the first user feature and the second user feature of each visiting user respectively. If there is a similarity greater than a similarity threshold, it is determined that there is an associated user of the target user among the visiting users. The similarity threshold is used to indicate the minimum value that the similarity between the user features of the target user and its associated users needs to meet.

[0118] Similarity is determined based on vectors. Specifically, the first and second user features are converted into corresponding feature vectors. Based on these feature vectors, the similarity between the first user feature and the second user feature of each visiting user is determined. If a similarity exceeds a similarity threshold, the visiting user is determined to be an associated user of the target user. The visiting user with a similarity greater than the similarity threshold is then identified as an associated user of the target user.

[0119] Another method is to cluster the target user and the visiting user based on the first user feature and the second user feature. If the visiting user and the target user are clustered into the same cluster, it is determined that there is an associated user of the target user among the visiting users.

[0120] If a visiting user and a target user are clustered into the same cluster, it means that the visiting user and the target user are highly similar. Therefore, it is determined that there are associated users of the target user among the visiting users, and the visiting users classified into the same cluster as the target user are determined as associated users of the target user.

[0121] At least one of the above two methods can be selected based on business needs. When both are selected, the crawler recognition results of the two are mutually verified.

[0122] 211. Perform anti-crawler operations on the associated users and end the current process.

[0123] The associated user is associated with the target user and is most likely a crawler. Therefore, an anti-crawler operation is performed on the associated user. The specific process of performing the anti-crawler operation can be found in the detailed description of step 212 below and will not be repeated here.

[0124] Furthermore, considering that some crawlers are used in compliance with regulations, and these compliant users cannot be banned (for example, they have purchased a designated membership from a data provider, and the designated membership is allowed to use crawlers), before performing anti-crawler operations on associated users, it is also necessary to determine whether the associated users are included in the users exempted from anti-crawler operations indicated in the whitelist. If not, perform the steps of performing anti-crawler operations on the associated users. If included, do not perform anti-crawler operations on the associated users.

[0125] 212. If the anti-crawler result indicates that an anti-crawler operation is to be performed on the target user, a target anti-crawler operation applicable to the target user is selected and the target anti-crawler operation is performed on the target user.

[0126] The target anti-crawler operation can be selected based on the user identification feature of the target user or based on the personalized designation of the target object, which is not limited in this embodiment. The target anti-crawler operation includes unconditional operation and conditional operation.

[0127] The unconditional operation is to ban the target user as long as the anti-crawler result indicates that the anti-crawler operation should be performed on the target user. The ban can be short-term or long-term, which can be selected by the target object.

[0128] Conditional operations involve the anti-crawler result indicating that an anti-crawler operation should be performed on the target user. A corresponding conditional judgment is then performed based on the conditional judgment result. The target anti-crawler operation includes at least one of the following operations: A to B:

[0129] Operation A: Determine whether the target user has been banned for a short period of time, and whether the number of short-term bans has reached a threshold. If so, permanently ban the target user; otherwise, ban the target user for a short period of time.

[0130] If it is determined that the target user has been banned for a short period of time, and the number of short-term bans has reached the threshold, it means that the target user is most likely a malicious crawler. Therefore, in order to reduce the harm of malicious crawlers, the target user will be permanently banned.

[0131] If it is determined that the target user has not been banned for a short period of time, or if it is determined that the target user has been banned for a short period of time and the number of short-term bans has not reached the threshold, it means that the target user has not caused significant harm, so the target user is banned for a short period of time.

[0132] Operation B is performed on all new accesses initiated by the target user to the target object: intercepting the response data of the target object to the access, desensitizing at least part of the response data, and feeding back the desensitized response data to the target user.

[0133] This operation does not ban the target user, but intercepts the target object's response data for access, desensitizes at least part of the response data, and feeds the desensitized response data back to the target user. In this way, even if the data is received by the target user, he or she will not be able to abuse the data.

[0134] Embodiments of the present application Figure 2 In the anti-crawler method provided, the detailed explanation of each step in the execution process can be found in the above Figure 1 The corresponding detailed explanation of the anti-crawler method embodiment will not be repeated here.

[0135] Furthermore, an embodiment of the present application provides an anti-crawler device, such as Figure 3 As shown, the anti-crawler device provided in this embodiment may include at least:

[0136] A first determining module 31 is configured to determine a target access parameter value in a real-time access log of a target user to a target object;

[0137] A second determining module 32 is configured to determine a target time period based on the target access occurrence time point corresponding to the real-time access log;

[0138] A third determining module 33 is configured to determine the access behavior characteristics and access parameter characteristics of the target user based on a target access log in a database where the access time point is within the target time period, wherein the database is configured to continuously collect the target user's access logs to the target object;

[0139] an identification module 34 for performing crawler identification on the target user based on the target access parameter value, the access behavior characteristics, and the access parameter characteristics;

[0140] The generating module 35 is used to generate an anti-crawler result for indicating whether to perform an anti-crawler operation on the target user based on the crawler identification result.

[0141] The anti-crawl device provided in an embodiment of the present application, after identifying a target user, determines the target access parameter value in the target user's real-time access log to the target object. It then determines a target time period based on the target access occurrence time point corresponding to the real-time access log, and determines the target user's access behavior characteristics and access parameter characteristics based on the target access log in the database whose access occurrence time point falls within the target time period. It then performs crawler identification on the target user based on the target access parameter value, access behavior characteristics, and access parameter characteristics. Finally, based on the crawler identification results, it generates an anti-crawl result indicating whether to perform an anti-crawl operation on the target user. Thus, the solution provided in this embodiment can achieve at least the following three effects: First, when performing an anti-crawl operation, the solution provided in this embodiment determines instantaneous characteristics such as access parameters in the user's real-time access log at discrete time points, as well as long-term behavioral characteristics such as access behavior characteristics and access parameter characteristics over the time series of the target time period. Finally, it combines these two dimensions of characteristics to perform crawler identification on the user, and generates an anti-crawl result for the target user based on the crawler identification results. In this way, since the anti-crawler operation is performed by integrating the features of the two dimensions, the missed detection rate and false positive rate of the anti-crawler can be reduced. Secondly, the solution provided by this embodiment does not require the intervention of the target user who accesses the target object when performing the anti-crawler operation. The entire anti-crawler operation is imperceptible to the target user, thus reducing the negative impact of the anti-crawler operation on the user experience. Thirdly, when performing the anti-crawler operation, the solution provided by this embodiment is based on the target user's access log to the target object. There is no need to encrypt the business content in the target object for user access, nor is there any need to obfuscate the code of the target object. Therefore, the anti-crawler operation provided by this embodiment will not affect the business content and code of the target object itself.

[0142] In some embodiments of the present application, Figure 4 As shown, the database is also used to continuously collect access logs of other users who access the target object except the target user. The anti-crawler device provided in this embodiment may also include:

[0143] a fourth determining module 36 configured to determine, if the anti-crawler result generated by the generating module 35 indicates that an anti-crawler operation is to be performed on the target user, a first user feature of the target user and a second user feature of the visiting user based on their respective access logs in the database;

[0144] The first judgment module 37 is configured to perform an anti-crawler operation on the associated user if it is determined that there is an associated user of the target user among the visiting users based on the first user feature of the target user and the second user feature of the visiting user.

[0145] In some embodiments of the present application, Figure 4 As shown, the first judgment module 37 may include at least one of the following: a first judgment unit 371 and a second judgment unit 372 .

[0146] The first judgment unit 371 is used to determine the similarity between the first user feature and the second user feature of each visiting user respectively. If there is a similarity greater than a similarity threshold, it is determined that there is an associated user of the target user among the visiting users. The similarity threshold is used to indicate the minimum value that the similarity between the user features of the target user and its associated users must meet.

[0147] The second judgment unit 372 is configured to cluster the target user and the visiting user based on the first user feature and the second user feature, and determine that there is an associated user of the target user among the visiting users if the visiting user and the target user are clustered into the same cluster.

[0148] The first user feature and the second user feature at least include at least one of the following features: access behavior feature, access parameter feature, user identification feature, and access parameter value.

[0149] In some embodiments of the present application, Figure 4 As shown, the identification module 34 may include:

[0150] A first identification unit 341 is configured to perform crawler identification on the target user based on the target access parameter value to obtain a first crawler identification result;

[0151] A second identification unit 342 is configured to perform crawler identification on the target user based on the access behavior characteristics and the access parameter characteristics to obtain a second crawler identification result;

[0152] The third recognition unit 343 is configured to determine a crawler recognition result for the target user based on the first crawler recognition result and the second crawler recognition result.

[0153] In some embodiments of the present application, Figure 4 As shown, the first identification unit 341 is specifically used to obtain the parameter threshold and target parameter value attribute corresponding to each target access parameter value at the time point when the target access occurs. The parameter threshold and target parameter value attribute are obtained based on the access log of the target object by normal users, and are dynamically updated as time goes by; for each target access parameter value, the following are performed: comparing the target access parameter value with its corresponding parameter threshold, and comparing the parameter value attribute of the target access parameter value with its corresponding target parameter value attribute to obtain the comparison result corresponding to the target access parameter value; based on the comparison result corresponding to each target access parameter value, a first crawler identification result is obtained.

[0154] In some embodiments of the present application, Figure 4 As shown, the second identification unit 342 is specifically configured to determine the normal access behavior characteristics and normal access parameter characteristics corresponding to the normal users of the target object, and obtain a second crawler identification result based on the similarity between the access behavior characteristics of the target object and the normal access behavior characteristics, as well as the similarity between the access parameter characteristics of the target object and the normal access parameter characteristics;

[0155] And / or, the second identification unit 342 is specifically used to cluster the target user and the normal user based on the normal access behavior characteristics and normal access parameter characteristics corresponding to the normal users of the target object, as well as the access behavior characteristics and access parameter characteristics of the target user, and obtain the second crawler identification result based on the clustering processing result.

[0156] In some embodiments of the present application, Figure 4 As shown, the first crawler identification result is used to indicate the probability that the target user is a crawler, and the second crawler identification result is used to indicate the probability that the target user is a crawler, then, the third identification unit 343 is specifically used to perform weighted calculation on the probability indicated by the first crawler identification result and the probability indicated by the second crawler identification result based on their respective corresponding confidence levels; if the weighted calculation result is greater than the probability threshold, a crawler identification result indicating that the target user is a crawler is obtained; if the weighted calculation result is not greater than the probability threshold, a crawler identification result indicating that the target user is a non-crawler is obtained.

[0157] In some embodiments of the present application, Figure 4 As shown, the generation module 35 is specifically used to send verification information to the target user if the crawler identification result indicates that the target user is a crawler, and the verification information is used to instruct the target user to perform the target action, and the target action is an action that the crawler is unable to perform; if the target information triggered by the execution of the target action is received, an anti-crawler result is generated to indicate that the anti-crawler operation is not performed on the target user; if the target information triggered by the execution of the target action is not received, an anti-crawler result is generated to indicate that the anti-crawler operation is performed on the target user.

[0158] and / or,

[0159] Generation module 35 is specifically used to detect whether the response data of the target object to the target user's access within the target time period carries target sensitive data, and the amount of target sensitive data carried is greater than the target threshold if the crawler identification result indicates that the target user is a crawler; if so, generate an anti-crawler result for indicating that an anti-crawler operation is performed on the target user; if not, generate an anti-crawler result for indicating that an anti-crawler operation is not performed on the target user.

[0160] and / or,

[0161] Generation module 35 is specifically used to detect whether the target user's access to the target object within the target time period affects the performance of the target object if the crawler identification result indicates that the target user is a crawler; if so, generate an anti-crawler result for indicating that an anti-crawler operation is performed on the target user; if not, generate an anti-crawler result for indicating that an anti-crawler operation is not performed on the target user.

[0162] In some embodiments of the present application, Figure 4 As shown, the anti-crawler device provided in this embodiment may also include:

[0163] The operation module 38 is used to select the target anti-crawler operation applicable to the target user and perform the target anti-crawler operation on the target user if the anti-crawler result generated by the generation module 35 indicates that an anti-crawler operation is to be performed on the target user; wherein, the target anti-crawler operation includes: determining whether the target user has been banned for a short period of time, and the number of short-term bans has reached a threshold number, and if so, permanently banning the target user; if not, banning the target user for a short period of time; or, for all new visits initiated by the target user to the target object, respectively executing: intercepting the response data of the target object to the visit, desensitizing at least part of the response data, and feeding back the desensitized response data to the target user.

[0164] In some embodiments of the present application, Figure 4 As shown, the second determination module 32 is specifically used to determine the starting time point and the ending time point based on the target access occurrence time point; determine the time period from the starting time point to the ending time point as the target time period; wherein, the starting time point is before the target access occurrence time point, and the ending time point is any one of the following: the ending time point is before the target access occurrence time point and after the starting time point, the ending time point is after the target access occurrence time point, and the ending time point is the target access occurrence time point.

[0165] In some embodiments of the present application, Figure 4 As shown, the anti-crawler device provided in this embodiment may also include:

[0166] The second judgment module 39 is used to determine the target user identification feature of the target user in the real-time access log; determine whether the target user identification feature matches the user identification feature of the crawler indicated by the blacklist; if so, the generation module 35 is triggered to generate a crawler identification result indicating that the target user is a crawler; if not, the first determination module 31 is triggered to execute the step of determining the target access parameter value in the real-time access log of the target user to the target object.

[0167] In the anti-crawler device provided in the embodiment of the present application, the detailed explanations used during the operation of each functional module can be found in the corresponding detailed explanations of the above-mentioned anti-crawler method embodiment, and will not be repeated here.

[0168] Furthermore, an embodiment of the present application also provides a computer-readable storage medium, which includes a stored program, wherein when the program is running, the device where the storage medium is located is controlled to execute the above-mentioned anti-crawler method.

[0169] Furthermore, an embodiment of the present application also provides an electronic device, which includes: a memory for storing a program; and a processor coupled to the memory for running the program to execute the above-mentioned anti-crawler method.

[0170] Furthermore, an embodiment of the present application also provides a computer program product, which includes: a computer program / computer executable instructions, and the computer program / computer executable instructions implement the above-mentioned anti-crawler method when executed by a processor.

[0171] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0172] It is understood that the relevant features of the above methods and devices can be referenced to each other. In addition, the terms "first" and "second" in the above embodiments are used to distinguish between the embodiments, and do not represent the advantages and disadvantages of the embodiments.

[0173] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0174] The algorithm and display provided herein are not inherently related to any particular computer, virtual system or other device. Various general-purpose systems can also be used together with the teachings based on this. According to the above description, it is obvious that the structure required for constructing such systems. In addition, the application is not directed to any specific programming language. It should be understood that various programming languages can be utilized to implement the content of the application described herein, and the above description of specific languages is for the purpose of disclosing the preferred embodiment of the application.

[0175] In addition, the memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0176] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0177] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data cutover device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data cutover device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0178] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data switching device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0179] These computer program instructions can also be loaded onto a computer or other programmable data switching device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable device to implement the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0180] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0181] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0182] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0183] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0184] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0185] The above are merely embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should all be included within the scope of the claims of the present application.

Claims

1. An anti-crawler method, characterized in that: The method comprises: Determine the target access parameter value in the real-time access log of the target user to the target object; Determine a target time period based on the target access occurrence time point corresponding to the real-time access log; Determining the access behavior characteristics and access parameter characteristics of the target user based on the target access log in the database where the access time point is within the target time period, wherein the database is used to continuously collect the access log of the target user to the target object; Performing crawler identification on the target user based on the target access parameter value, the access behavior characteristics, and the access parameter characteristics; Based on the crawler identification result, an anti-crawler result is generated to indicate whether to perform an anti-crawler operation on the target user.

2. The method according to claim 1, characterized in that The database is further configured to continuously collect access logs of other users accessing the target object other than the target user. The method further comprises: If the anti-crawler result indicates that an anti-crawler operation is to be performed on the target user, determining a first user feature of the target user and a second user feature of the visiting user based on their respective access logs in the database; If it is determined based on the first user feature of the target user and the second user feature of the visiting user that there is a user associated with the target user among the visiting users, an anti-crawler operation is performed on the associated user.

3. The method according to claim 2, characterized in that Determining, based on a first user feature of the target user and a second user feature of the visiting user, whether there is an associated user of the target user among the visiting users includes: Determining the similarity between the first user feature and the second user feature of each visiting user, respectively; if the similarity is greater than a similarity threshold, determining that there is an associated user of the target user among the visiting users, wherein the similarity threshold is used to indicate a minimum value that the similarity between the user features of the target user and its associated users must meet; and / or, clustering the target user and the visiting user based on the first user feature and the second user feature, and if the visiting user and the target user are clustered into the same cluster, determining that there is a user associated with the target user among the visiting users; The first user feature and the second user feature at least include at least one of the following features: access behavior feature, access parameter feature, user identification feature, and access parameter value.

4. The method according to claim 1, wherein Performing crawler identification on the target user based on the target access parameter value, the access behavior characteristics, and the access parameter characteristics, including: Perform crawler identification on the target user based on the target access parameter value to obtain a first crawler identification result; Perform crawler identification on the target user based on the access behavior characteristics and the access parameter characteristics to obtain a second crawler identification result; A crawler recognition result for the target user is determined based on the first crawler recognition result and the second crawler recognition result.

5. The method according to claim 4, characterized in that Performing crawler identification on the target user based on the target access parameter value to obtain a first crawler identification result includes: Obtaining the parameter threshold and target parameter value attributes corresponding to each target access parameter value at the time point when the target access occurs, where the parameter threshold and target parameter value attributes are obtained based on the access log of the target object by normal users and are dynamically updated over time; For each target access parameter value, respectively performing the following steps: comparing the target access parameter value with its corresponding parameter threshold, and comparing the parameter value attribute of the target access parameter value with its corresponding target parameter value attribute, to obtain a comparison result corresponding to the target access parameter value; Based on the comparison results corresponding to each target access parameter value, a first crawler identification result is obtained.

6. The method according to claim 4, characterized in that Performing crawler identification on the target user based on the access behavior characteristics and the access parameter characteristics to obtain a second crawler identification result, including: Determining normal access behavior characteristics and normal access parameter characteristics corresponding to normal users of the target object, and obtaining a second crawler recognition result based on the similarity between the access behavior characteristics of the target object and the normal access behavior characteristics, and the similarity between the access parameter characteristics of the target object and the normal access parameter characteristics; And / or, based on the normal access behavior characteristics and normal access parameter characteristics corresponding to normal users of the target object, and the access behavior characteristics and access parameter characteristics of the target user, the target user and the normal user are clustered, and a second crawler identification result is obtained based on the clustering processing result.

7. The method according to claim 4, characterized in that The first crawler identification result is used to indicate the probability that the target user is a crawler, and the second crawler identification result is used to indicate the probability that the target user is a crawler. Then, based on the first crawler identification result and the second crawler identification result, determining the crawler identification result for the target user includes: Based on the confidence levels corresponding to the first crawler identification result and the second crawler identification result, performing a weighted calculation on the probability indicated by the first crawler identification result and the probability indicated by the second crawler identification result; If the weighted calculation result is greater than the probability threshold, a crawler identification result is obtained, indicating that the target user is a crawler; If the weighted calculation result is not greater than the probability threshold, a crawler identification result is obtained, indicating that the target user is a non-crawler.

8. The method according to claim 1, characterized in that Based on the crawler identification result, an anti-crawler result is generated for indicating whether to perform an anti-crawler operation on the target user, including: if the crawler identification result indicates that the target user is a crawler, sending verification information to the target user, the verification information is used to instruct the target user to perform a target action, and the target action is an action that the crawler is unable to perform; if target information caused by the execution of the target action is received, generating an anti-crawler result for indicating not to perform the anti-crawler operation on the target user; if the target information caused by the execution of the target action is not received, generating an anti-crawler result for indicating to perform the anti-crawler operation on the target user; and / or, Based on the crawler identification result, an anti-crawler result is generated for indicating whether to perform an anti-crawler operation on the target user, including: if the crawler identification result indicates that the target user is a crawler, detecting whether the response data of the target object to the target user's access within the target time period carries target sensitive data, and the amount of target sensitive data carried is greater than a target threshold; if so, generating an anti-crawler result for indicating that the anti-crawler operation is performed on the target user; if not, generating an anti-crawler result for indicating that the anti-crawler operation is not performed on the target user; and / or, Based on the crawler identification result, an anti-crawler result is generated to indicate whether an anti-crawler operation is performed on the target user, including: if the crawler identification result indicates that the target user is a crawler, detecting whether the target object's access to the target user within the target time period affects the performance of the target object; if so, generating an anti-crawler result to indicate that an anti-crawler operation is performed on the target user; if not, generating an anti-crawler result to indicate that an anti-crawler operation is not performed on the target user.

9. The method according to any one of claims 1 to 8, characterized in that The method further includes: if the anti-crawler result indicates that an anti-crawler operation is to be performed on the target user, selecting a target anti-crawler operation applicable to the target user, and performing the target anti-crawler operation on the target user; wherein the target anti-crawler operation includes: determining whether the target user has been temporarily banned, and whether the number of short-term bans has reached a threshold, and if so, permanently banning the target user; if not, temporarily banning the target user; or, for all new accesses initiated by the target user to the target object, respectively performing the following: intercepting the response data of the target object to the access, desensitizing at least part of the response data, and feeding back the desensitized response data to the target user; and / or, Determining a target time period according to the target access occurrence time point corresponding to the real-time access log includes: determining a start time point and an end time point based on the target access occurrence time point; determining the time period from the start time point to the end time point as the target time period; wherein the start time point is before the target access occurrence time point, and the end time point is any one of the following: the end time point is before the target access occurrence time point and after the start time point, the end time point is after the target access occurrence time point, and the end time point is the target access occurrence time point; and / or, The method also includes: determining a target user identification feature of the target user in the real-time access log; judging whether the target user identification feature matches a user identification feature of a crawler indicated by a blacklist; if so, generating a crawler identification result indicating that the target user is a crawler; if not, executing a step of determining a target access parameter value in the real-time access log of the target user to the target object. 10.An anti-crawler device, characterized in that: The device comprises: A first determining module is used to determine a target access parameter value in a real-time access log of a target user to a target object; A second determining module is configured to determine a target time period according to a target access occurrence time point corresponding to the real-time access log; a third determining module, configured to determine the access behavior characteristics and access parameter characteristics of the target user based on a target access log in a database where the access time point is within the target time period, wherein the database is configured to continuously collect the target user's access log to the target object; an identification module, configured to perform crawler identification on the target user based on the target access parameter value, the access behavior characteristics, and the access parameter characteristics; A generation module is used to generate an anti-crawler result based on the crawler identification result, which is used to indicate whether to perform an anti-crawler operation on the target user.

11. A computer-readable storage medium, characterized in that The storage medium includes a stored program, wherein when the program is running, the device where the storage medium is located is controlled to execute the anti-crawler method according to any one of claims 1 to 9.

12. An electronic device, characterized in that: The electronic device includes: a memory for storing a program; and a processor coupled to the memory for running the program to execute the anti-crawler method according to any one of claims 1 to 9.

13. A computer program product, characterized in that The computer program product includes: a computer program / computer executable instructions, which, when executed by a processor, implements the anti-crawler method described in any one of claims 1 to 9.

Images