Anti-crawler processing method, device, electronic device and readable storage medium

By finding the crawler index of the target IP from the IP access records of multiple website nodes and performing anti-crawler operations above the threshold, the crawler IP recognition problem is solved, and data security and identification efficiency are improved.

CN115065520BActive Publication Date: 2024-11-12CHINA TELECOM CORP LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210649745.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-06-09
Publication Date
2024-11-12
Estimated Expiration
2042-06-09

AI Technical Summary

Technical Problem

The prior art is difficult to quickly identify crawler IP and perform anti-crawler operations, which makes it difficult to solve the problem of illegal crawling data.

Method used

By finding the crawler index of the target IP from the IP access records of multiple website nodes, if the index is greater than the preset threshold, an anti-crawler operation is performed, and blockchain technology is used to share and maintain the crawler index to improve identification accuracy and efficiency.

Benefits of technology

It realizes rapid identification and effective defense of crawler IP, improves data security, and reduces identification errors and processing costs.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115065520B_ABST
    Figure CN115065520B_ABST
Patent Text Reader

Abstract

The embodiment of the present invention provides an anti-crawler processing method, device, electronic device and readable storage medium. In the method, in response to an access request, the crawler index corresponding to the target IP is searched from the crawler index corresponding to the first historical IP as the target crawler index; the first historical IP and the crawler index corresponding to the first historical IP are obtained according to the IP access records of at least two second website nodes, and the crawler index is used to characterize the probability that the first historical IP belongs to the crawler IP, and the target IP is the IP used for the access request. When the target crawler index is greater than the preset index threshold, the preset anti-crawler operation is performed. In this way, by using the access records of multiple second website nodes to identify the crawler IP, it is possible to quickly identify the crawler IP and perform anti-crawler operations on it.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of network technology, and in particular relates to an anti-crawler processing method, device, electronic device and readable storage medium. Background Art

[0002] With the continuous development of network technology, more and more websites provide data for users to access. Users can use Internet Protocol Address (IP) to access the website to view the data provided by the website.

[0003] However, in actual applications, there are illegal users who illegally obtain data provided by websites based on crawler technology. Therefore, how to quickly identify crawler IPs, that is, identify the IPs used when illegally crawling data based on crawler technology, and perform anti-crawler operations on them, has become a technical problem that needs to be solved urgently. Summary of the invention

[0004] The present invention provides an anti-crawler processing method, device, electronic device and readable storage medium to solve the technical problem of how to quickly identify crawler IPs and perform anti-crawler operations on them.

[0005] In a first aspect, the present invention provides an anti-crawler processing method, which is applied to a first website node, and the method comprises:

[0006] In response to the access request, searching for a crawler index corresponding to a target IP from the crawler index corresponding to the first historical Internet Protocol address IP as a target crawler index; the first historical IP and the crawler index corresponding to the first historical IP are obtained according to the IP access records of at least two second website nodes, the crawler index is used to characterize the probability that the first historical IP belongs to a crawler IP, and the target IP is the IP used for the access request;

[0007] When the target crawler index is greater than a preset index threshold, a preset anti-crawler operation is performed.

[0008] In a second aspect, the present invention provides an anti-crawler processing device, applied to a first website node, the device comprising:

[0009] a search module, configured to search for a crawler index corresponding to a target IP from the crawler index corresponding to the first historical Internet Protocol address IP in response to an access request, as a target crawler index; the first historical IP and the crawler index corresponding to the first historical IP are obtained according to IP access records of at least two second website nodes, the crawler index is used to characterize the probability that the first historical IP belongs to a crawler IP, and the target IP is the IP used by the access request;

[0010] The execution module is used to execute a preset anti-crawler operation when the target crawler index is greater than a preset index threshold.

[0011] In a third aspect, the present invention provides an anti-crawler processing system, the system comprising at least two first website nodes, each of which is used to perform the following steps:

[0012] In response to the access request, searching the crawler index corresponding to the target IP from the crawler index corresponding to the first historical IP as the target crawler index; the first historical IP and the crawler index corresponding to the first historical IP are obtained according to the IP access records of the at least two first website nodes, the crawler index is used to characterize the probability that the first historical IP belongs to the crawler IP, and the target IP is the IP used by the access request;

[0013] When the target crawler index is greater than a preset index threshold, a preset anti-crawler operation is performed.

[0014] In a fourth aspect, the present invention provides an electronic device comprising: a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned anti-crawler processing method when executing the program.

[0015] In a fifth aspect, the present invention provides a readable storage medium, which, when the instructions in the storage medium are executed by a processor of an electronic device, enables the electronic device to execute the above-mentioned anti-crawler processing method.

[0016] In an embodiment of the present invention, in response to an access request, the crawler index corresponding to the target IP is searched from the crawler index corresponding to the first historical IP as the target crawler index; the first historical IP and the crawler index corresponding to the first historical IP are obtained based on the IP access records of at least two second website nodes, and the crawler index is used to characterize the probability that the first historical IP belongs to the crawler IP, and the target IP is the IP used in the access request. When the target crawler index is greater than the preset index threshold, a preset anti-crawler operation is performed. In this way, the crawler IP is identified using the access records of multiple second website nodes, providing more sufficient information for crawler IP identification. The first website node uses the first historical IP obtained based on the IP access records of at least two second website nodes and its corresponding crawler index for search, and can identify the crawler IP, thereby achieving rapid identification of the crawler IP to a certain extent, and performing anti-crawler operations on it.

[0017] At the same time, compared with the method of identifying based only on the historical data of the website itself, even when the target IP visits the first website node for the first time, since the target IP may have visited a second website node before, the first website node uses the crawler index obtained according to the IP access records of at least two second website nodes for identification, which can increase the probability of finding the crawler index of the target IP to a certain extent, so that when the target IP visits for the first time, it can quickly identify and implement anti-crawler measures in time. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1 It is a flowchart of the steps of an anti-crawler processing method provided by an embodiment of the present invention;

[0020] Figure 2 is a schematic diagram of a scenario provided by an embodiment of the present invention;

[0021] Figure 3 is a structural diagram of an anti-crawler processing device provided by an embodiment of the present invention;

[0022] Figure 4 It is a structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0023] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0024] Figure 1 is a flowchart of the steps of an anti-crawler processing method provided by an embodiment of the present invention, wherein the method is applied to a first website node, such as Figure 1 As shown, the method may include:

[0025] Step 101, in response to an access request, search the crawler index corresponding to the target IP from the crawler index corresponding to the first historical IP as the target crawler index; the first historical IP and the crawler index corresponding to the first historical IP are obtained based on the IP access records of at least two second website nodes, the crawler index is used to characterize the probability that the first historical IP belongs to the crawler IP, and the target IP is the IP used for the access request.

[0026] In an embodiment of the present invention, the first website node can parse the access request and extract the IP used by the access request from the access request as the target IP. The target IP is matched with each first historical IP. If the match is successful, the crawler index corresponding to the matched first historical IP is determined as the target crawler index. The crawler index can be used to characterize the probability that the IP belongs to the crawler IP. The higher the crawler index, the greater the probability that the IP belongs to the crawler IP, and the smaller the crawler index, the smaller the probability that the IP belongs to the crawler IP.

[0027] The IP access record may be an IP access log, and the IP access log may record the relevant information of the IP used in the access request for accessing the second website node. The first historical IP involved in the crawler index corresponding to the first historical IP may be an IP that has visited the second website node, and the crawler index of the first historical IP may be generated by integrating the IP access records of multiple second website nodes.

[0028] The first website node and the second website node may be backend servers of the website. Different second website nodes may correspond to different websites.

[0029] Step 102: When the target crawler index is greater than a preset index threshold, perform a preset anti-crawler operation.

[0030] The preset index threshold can be set according to actual needs. If the target crawler index is greater than the preset index threshold, the target IP can be determined to be a crawler IP, and the preset anti-crawler operation can be performed. The preset index thresholds of different website nodes can be set according to actual needs, and the preset index thresholds of different website nodes can be different or the same.

[0031] The preset anti-crawler operation can be set according to time requirements. For example, the preset anti-crawler operation may include adding the target IP to an access blacklist, returning a prompt message indicating prohibited access to the sender of the access request, adding verification code verification, blocking IP, etc., to prevent the sender from obtaining data from the website represented by the first website node based on the target IP.

[0032] Furthermore, when the target crawler index is not greater than the preset index threshold, it can be determined that the target IP is not a crawler IP, and accordingly, the data in the website represented by the first website node can be returned to the sender, so that the sender user can normally access the website represented by the first website node, and normally obtain and view the data in the website. It should be noted that if the target crawler index corresponding to the target IP is not found, the target IP's current access operation can be allowed first, and the data in the website represented by the first website node is returned to the sender. When the IP access record is updated, resulting in the crawler index corresponding to the first historical IP being updated, and then the crawler index corresponding to the first historical IP covers the crawler index of the target IP, when the target IP visits the first website node again, it can be searched again based on the crawler index corresponding to the first historical IP.

[0033] In summary, the anti-crawler processing method provided by the embodiment of the present invention, in response to an access request, searches for the crawler index corresponding to the target IP from the crawler index corresponding to the first historical Internet Protocol address IP as the target crawler index; the first historical IP and the crawler index corresponding to the first historical IP are obtained according to the IP access records of at least two second website nodes, and the crawler index is used to characterize the probability that the first historical IP belongs to the crawler IP, and the target IP is the IP used for the access request. When the target crawler index is greater than the preset index threshold, the preset anti-crawler operation is performed. In this way, the crawler IP is identified by using the access records of multiple second website nodes to provide more sufficient information for crawler IP identification. The first website node uses the first historical IP obtained according to the IP access records of at least two second website nodes and its corresponding crawler index for search, and can identify the crawler IP, thereby achieving rapid identification of the crawler IP to a certain extent, and performing anti-crawler operations on it.

[0034] At the same time, compared with the method of identifying based only on the historical data of the website itself, even when the target IP visits the first website node for the first time, since the target IP may have visited a second website node before, the first website node uses the crawler index obtained according to the IP access records of at least two second website nodes for identification, which can increase the probability of finding the crawler index of the target IP to a certain extent, so that when the target IP visits for the first time, it can quickly identify and implement anti-crawler measures in time.

[0035] Optionally, in the embodiment of the present invention, the crawler index corresponding to the first historical IP is stored in the blockchain, and the first website node is any one of the at least two second website nodes, and any of the first website nodes is locally deployed with the blockchain. Specifically, compared with other non-second website nodes, the crawler index corresponding to the first historical IP can also be used. In the embodiment of the present invention, at least any one of the two second website nodes can be used to execute the anti-crawler processing method provided by the embodiment of the present invention, that is, at least any one of the two second website nodes can perform crawler IP identification based on the blockchain storing the crawler index corresponding to the first historical IP, thereby improving the security of the data to a certain extent. In other words, at least two second website nodes can form an alliance chain, and each website node participating in the alliance chain can perform crawler identification based on the crawler index on the chain. The alliance chain is essentially a blockchain with high robustness and can ensure stability. And compared with the way that other non-second website nodes can also access it as a public chain, the use of the alliance chain can reduce the use cost of participating nodes to a certain extent, improve controllability and data security, and because no verification is required, the transaction speed on the chain can be improved to a certain extent.

[0036] Accordingly, since the first website node is locally deployed with a blockchain, when performing a search operation, the first website node can directly search based on the crawler index corresponding to the first historical IP recorded in the local blockchain, thereby improving the search efficiency to a certain extent.

[0037] It should be noted that IPs in the blockchain whose crawler index is greater than the preset index threshold can be blacklisted to facilitate searching.

[0038] Optionally, the following steps may also be performed in the embodiment of the present invention:

[0039] Step S21, when the preset conditions are met, determine the stage crawler index of the second historical IP within the preset time according to the IP access record of the first website node within the preset time; the stage crawler index is used to characterize the probability that the second historical IP belongs to a crawler within the preset time.

[0040] In an embodiment of the present invention, the preset condition and the preset time can be set according to actual needs. Exemplarily, the preset condition can be to reach a preset execution time interval, that is, this step can be performed multiple times, so as to realize dynamic maintenance of the crawler index. Or the preset condition can also be receiving a calculation instruction, etc. The preset time can be the last 3 days, the last 7 days, etc. Correspondingly, after each interval of the preset time, according to the IP access record within the preset time, the stage crawler index of the second historical IP within the preset time is determined once. Among them, the second historical IP can be an IP that accesses the first website node within the preset time, and the second historical IP can belong to the aforementioned first historical IP. For example, after the crawler index of the second historical IP is chained, the second historical IP can be regarded as the first historical IP. The stage crawler index can be used to characterize the probability that the second historical IP belongs to the crawler within the preset time, and accordingly, the crawler index of the second historical IP can be used to characterize the probability that the second historical IP currently belongs to the crawler. That is to say, the stage crawler index is the probability of the IP belonging to the crawler from a certain period of time, and the crawler index is the probability of the IP belonging to the crawler from the overall evaluation. For example, the stage crawler index can be represented by P, and the crawler index can be represented by P'. The crawler index can represent the final crawler index.

[0041] Step S22, linking the stage crawler index to the blockchain in the first website node, and, based on the stage crawler index and the historical stage crawler index of the second historical IP, generating a new crawler index for the second historical IP, and linking the new crawler index to the blockchain in the first website node; the historical stage crawler index includes the stage crawler index previously linked to the blockchain.

[0042] In an embodiment of the present invention, after the stage crawler index is chained, a new crawler index can be generated for the second historical IP based on the stage crawler index and the historical stage crawler index of the second historical IP, and the new crawler index can be chained to the blockchain in the first website node. Alternatively, a new crawler index can be generated for the second historical IP based on the stage crawler index and the historical stage crawler index of the second historical IP, and the new crawler index can be chained to the blockchain in the first website node, and then the stage crawler index can be chained. Alternatively, a new crawler index can be generated for the second historical IP based on the stage crawler index and the historical stage crawler index of the second historical IP, and the new crawler index and the stage crawler index can be chained together to the blockchain in the first website node.

[0043] Specifically, the second historical IP and the crawler index of this stage can be packaged into a block, and the block can be chained, so as to realize the chaining of the stage crawler index to the local blockchain of the first website node. The second historical IP and the new crawler index are packaged into a block, and the chaining operation is performed on the block, so as to realize the chaining of the new crawler index to the local blockchain of the first website node. It should be noted that other information can be further packaged into the block during packaging, for example, the second historical IP, the crawler index of this stage, and the timestamp of the crawler index of this stage are packaged into a block. The second historical IP, the new crawler index, and the timestamp of the new crawler index are packaged into a block. Among them, the timestamp can be used to characterize the generation time of the stage crawler index / crawler index.

[0044] It should be noted that the execution order of the above steps S21 to S22 is not unique. For example, steps S21 to S22 can be executed before or after step 101, or after step 102, and the embodiment of the present invention does not limit this. Steps S21 to S22 can be repeated multiple times to achieve dynamic maintenance of the crawler index.

[0045] As time goes by, the nature of the IP may change. For example, a certain IP may be used as a crawler IP from a certain time. And a previous crawler IP may be released and used as an IP for normal users. In the embodiment of the present invention, any first website node regularly determines the stage crawler index of the second historical IP based on the IP access record within a preset time, and generates a new crawler index based on the stage crawler index of the second historical IP and the previous stage crawler index. In this way, the crawler index of the second historical IP can accurately characterize the probability that the second historical IP is currently a crawler IP, thereby ensuring the accuracy of subsequent identification.

[0046] Optionally, in the embodiment of the present invention, after the step of determining the stage crawler index of the second historical IP within the preset time, the following steps may be performed:

[0047] Step S31, synchronize the stage crawler index to other first website nodes deployed with the blockchain, so that the other first website nodes can generate a new crawler index for the second historical IP based on the stage crawler index and the historical stage crawler index of the second historical IP, and upload the new crawler index and the received stage crawler index to the blockchain in the other first website nodes.

[0048] Specifically, other first website nodes may be: nodes other than the first website node that calculates the crawler index of this stage among multiple first website nodes. The stage crawler index may be broadcast to other first website nodes to achieve synchronization. Accordingly, each other first website node may adopt the same calculation method to generate a new crawler index for the second historical IP based on the received stage crawler index and the historical stage crawler index of the second historical IP. Among them, other first website nodes may read the historical stage crawler index of the second historical IP from the local blockchain. After calculating the new crawler index, the new crawler index is chained to the local blockchain. Furthermore, other first website nodes may also chain the received stage crawler index of the second historical IP to the blockchain in the other first website nodes. Among them, the execution order of the operation of the stage crawler index received on the chain and the operation of the new crawler index can be selected as needed.

[0049] In the embodiment of the present invention, the first website node only needs to synchronize the stage crawler index calculated by itself to other first website nodes deployed with blockchain, so that other first website nodes can upload the new crawler index of the second historical IP to the local blockchain, thereby ensuring the synchronization of the content of the blockchain held by each first website node, thereby ensuring the crawler identification effect of each first website node based on blockchain.

[0050] At the same time, other first website nodes generate new crawler indexes themselves and upload their generated new crawler indexes to the local blockchain, which can ensure the security of the data to a certain extent.

[0051] It should be noted that, in the embodiment of the present invention, the first website node can also synchronize the second historical IP and the calculated new crawler index to other first website nodes deployed with blockchain. In this way, other first website nodes do not need to perform the operation of generating a new crawler index, and can directly upload the received new crawler index to the chain, thereby saving processing costs.

[0052] Optionally, the following steps may also be performed in the embodiment of the present invention:

[0053] Step S41: If a stage crawler index of a third historical IP shared by other first website nodes is received, a new crawler index is generated for the third historical IP based on the stage crawler index shared by the other first website nodes and the historical stage crawler index of the third historical IP, and the new crawler index and the received stage crawler index are linked to the blockchain in the first website node.

[0054] In an embodiment of the present invention, other first website nodes may determine the stage crawler index of the third historical IP within the preset time according to the IP access record of the other first website nodes within the preset time when the preset conditions are met. Among them, the third historical IP may be the same as or different from the second historical IP, and the implementation method of determining the stage crawler index of the third historical IP within the preset time may be the same as the implementation method of determining the stage crawler index of the second historical IP within the preset time. Accordingly, the other first website node can link the stage crawler index to the blockchain in the first website node, and, based on the stage crawler index and the historical stage crawler index of the third historical IP, generate a new crawler index for the third historical IP. And link the new crawler index to the blockchain in the first website node, and share the stage crawler index with the first website nodes other than itself. Among them, the implementation method of generating a new crawler index for the third historical IP may be the same as the implementation method of generating a new crawler index for the second historical IP.

[0055] As the first website node that receives the shared stage crawler index, a new crawler index can be generated for the third historical IP based on the received stage crawler index and the historical stage crawler index of the third historical IP, and the third historical IP and the new crawler index can be packaged into a block, and the chain operation can be performed, thereby ensuring the synchronization of the content of the blockchain held by each first website node and ensuring the crawler recognition effect of each first website node based on the blockchain. Further, the third historical IP and the received stage crawler index of the third historical IP can be packaged into a block, and the chain operation can be performed.

[0056] In an embodiment of the present invention, different websites represented by different first website nodes may correspond to different applications. Multiple first website nodes determine the stage crawler index of the historical IP that accesses themselves based on their own IP access records within a preset time, and through sharing, each first website node can chain the newly generated crawler index of the historical IP in the local blockchain, thereby realizing the joint maintenance of the blockchain by multiple websites. Accordingly, crawler IP risk detection based on the crawler index can be regarded as crawler IP risk detection based on the blockchain jointly maintained by multiple websites. Among them, crawler IP risk detection based on the crawler index refers to: identifying whether the target IP is a crawler IP based on the found target crawler index.

[0057] Optionally, the operation of generating a new crawler index for the second historical IP according to the crawler index of the stage and the crawler index of the second historical IP in the historical stage may specifically include:

[0058] Step S51: Obtain the historical stage crawler index of the second historical IP from the chained blocks of the blockchain, and obtain the generation time of the historical stage crawler index as the first time.

[0059] In an embodiment of the present invention, a block for storing a stage crawler index may be provided with a first identifier, and a block for storing a crawler index may be provided with a second identifier, and the first identifier is different from the second identifier. Accordingly, if the identifier of the chained block in the local blockchain of the first website node is the first identifier and the historical IP stored in the chained block is the second historical IP, the stage crawler index in the chained block can be read as the historical stage crawler index of the second historical IP. Among them, the historical stage crawler index of the second historical IP may not exist, or may be one or more. As time goes by, the historical stage crawler index of the second historical IP may increase.

[0060] Furthermore, the block for storing the crawler index of the stage may also store the timestamp of the crawler index of the stage. Accordingly, the timestamp of the chained block with the first identifier and the stored historical IP as the second historical IP may be read as the first time.

[0061] Step S52: Generate a new crawler index for the second historical IP according to the historical stage crawler index, the first time, the stage crawler index and the second time; the second time is the generation time of the stage crawler index.

[0062] Among them, the generation time of the crawler index of this stage can be recorded when the crawler index of this stage is generated. As time goes by, the access behavior of the IP may change, that is, the IP may change from a normal IP to a crawler IP, or from a crawler IP to a normal IP. Therefore, in the embodiment of the present invention, the first time of the historical stage crawler index and the second time of the stage crawler index are obtained, and the historical stage crawler index, the first time, the stage crawler index and the second time are combined to generate a new crawler index for the second historical IP, which can ensure the accuracy of the generated crawler index to a certain extent.

[0063] Optionally, the operation of generating a new crawler index for the second historical IP according to the historical stage crawler index, the first time, the stage crawler index and the second time may specifically include:

[0064] Step S61, according to the first time and the second time, respectively determine the first weight of the historical stage crawler index and the second weight of the stage crawler index; the first weight is negatively correlated with the length of time from the first time to the current time, and the second weight is negatively correlated with the length of time from the second time to the current time.

[0065] In the embodiment of the present invention, if the generation time of the stage crawler index is closer to the current time, it can be determined that the stage crawler index can better represent the current situation of the second historical IP. Therefore, in this step, a negative correlation method can be used to determine the first weight for each historical stage crawler index according to the first time and the second time, and determine the second weight for the stage crawler index.

[0066] Among them, since the stage crawler index is generated recently, the second weight can be greater than any first weight. For multiple historical stage crawler indexes, the earlier the generation time, the smaller the first weight of the historical stage crawler index can be.

[0067] Exemplarily, the calculation may be performed based on a preset time decay function. X represents the weight, and the weight may be calculated based on the following formula:

[0068] X=e -α(t-t0)

[0069] Wherein, α represents a preset attenuation coefficient, t represents the current time, and t0 represents the generation time. t-t0 represents the distance between the generation time and the current time. For the first weight, t0 is the first time, and for the second weight, t0 is the second time. Of course, in actual application scenarios, other attenuation formulas can also be used for attenuation to calculate the weight, and this embodiment does not limit this.

[0070] Step S62: Generate the new crawler index according to the first weight, the second weight, the historical crawler index and the stage crawler index.

[0071] In an embodiment of the present invention, the product of the second weight and the crawler index of the stage can be calculated, and the product of the crawler index of each historical stage and the first weight of the crawler index of the historical stage can be calculated. Then, the new crawler index is determined according to the sum of the products. For example, the sum of the products is directly determined as the new crawler index. Accordingly, the new crawler index can be expressed as:

[0072] As time goes by, the earlier the crawler index is, the less weight it occupies in the final P'. Where n∈1,…,y. Pn represents the nth risk index involved in the calculation. Assuming that the first risk index involved in the calculation is the crawler index of this stage, then p1 represents the crawler index of this stage, and the remaining p2,…,py represent the crawler index of the historical stage. y represents the total number of risk indexes involved in the calculation. The total number refers to the sum of the number of crawler indexes in the historical stage and the number of crawler indexes in the stage, that is, the number of crawler indexes in the historical stage plus 1.

[0073] Alternatively, the ratio of the sum of the products to the total number may be determined as the new crawler index, which is not limited in the embodiment of the present invention.

[0074] In an embodiment of the present invention, weights are set accordingly according to the generation time, so that the risk index generated earlier has less impact on the crawler index finally generated, thereby ensuring the accuracy of the generated crawler index to a certain extent and reducing the impact of time on IP crawler risk judgment.

[0075] Correspondingly, the above operation of searching for the crawler index corresponding to the target IP from the crawler index corresponding to the first historical IP may specifically include: searching for the latest generated crawler index corresponding to the target IP from the blockchain of the first website node. In this step, the stored crawler index and timestamp can be read from the chained block marked with the second identifier and the stored historical IP as the target IP. Search for the earliest crawler index represented by the corresponding timestamp as the target crawler index. In an embodiment of the present invention, by searching for the latest generated crawler index corresponding to the target IP, it can be ensured that the crawler index found is the crawler index that best suits the current situation, thereby ensuring the accuracy of subsequent crawler identification and avoiding unnecessary execution of preset anti-crawler operations.

[0076] It should be noted that, for any first website node, the above-mentioned operations of determining the stage crawler index of the second historical IP within the preset time, uploading the stage crawler index to the blockchain in the first website node, generating a new crawler index for the second historical IP, and synchronizing the stage crawler index to other first website nodes deployed with blockchain can be performed multiple times.

[0077] Optionally, the operation of determining the stage crawler index of the second historical IP within the preset time according to the IP access record of the first website node within the preset time may specifically include:

[0078] Step S71: for a second historical IP within the preset time, determine a first usage count, an access count, and a second usage count of an access proxy corresponding to the second historical IP according to the IP access record.

[0079] The access source may refer to the referer parameter of the historical access request carrying the second historical IP. For example, the access source may be a part of the request header in the HTTP request, which is used to indicate the source of the client requesting the current resource. The access agent may refer to the user agent. The user agent may be a special string header, and the user agent may be the first-level identifier that indicates the identity.

[0080] For each historical access request, the IP used by the historical access request can be extracted as the second historical IP, and the corresponding access source and access proxy can be obtained and written into the IP access record. The implementation method of obtaining the access source and access proxy can be selected according to actual needs. For example, the access source and access proxy can be extracted from the request header of the historical request.

[0081] Specifically, the second historical IP that appears can be counted based on the IP access records within a preset time. There may be multiple records with the second historical IP for a second historical IP. Among them, one record can correspond to a historical access request, that is, one access. From the multiple records corresponding to the second historical IP, extract the access source and access agent recorded in each record. Multiple records may involve multiple access sources, and the number of occurrences of the most frequent access source is determined as the first number of uses. Suppose 10 access sources are extracted, of which 8 access sources are access source 1 and 2 access sources are access source 2. Then 8 can be used as the first number of uses. Multiple records may involve multiple access agents, and the number of occurrences of the most frequent access agent is determined as the second number of uses. Suppose 10 access agents are extracted, 7 of which are access 1 and 3 are access agent 2. Then 7 can be used as the second number of uses.

[0082] The number of visits corresponding to the second historical IP may refer to the total number of requests of the second historical IP within a preset time, and the number of multiple records corresponding to the second historical IP may be determined as the number of visits. The preset time may be set according to actual needs, and illustratively, the preset time may be 1 hour. The number of visits corresponding to the second historical IP may represent the access frequency of the second historical IP.

[0083] Step S72, calculate the stage crawler index of the second historical IP according to the first number of uses, the number of visits, the second number of uses and the total number of visits within the preset time; the stage crawler index is positively correlated with the first number of uses and the number ratio, and negatively correlated with the second number of uses, and the number ratio is the ratio of the number of visits to the total number of visits.

[0084] The total number of accesses within the preset time may be the total number of IP access records within the preset time. The first usage number may be represented as r, the second usage number may be represented as u, the access number may be represented as c, and the total number of accesses within the preset time may be represented as c'.

[0085] Exemplarily, the stage crawler index P of the second historical IP can be expressed as:

[0086]

[0087] Among them, Δr, Δu, Δc represent preset weights, and Δr+Δu+Δc=1.

[0088] For the same IP, if its access sources are more uniform, the probability of a crawler is greater. If the user agents that access are relatively dispersed, the probability of a crawler is greater. If the IP access times account for a large proportion, the probability of a crawler is greater. The larger the first usage times, it can be determined that the request sender frequently uses the same access source within the preset time. The smaller the first usage times, it can be determined that the access agents used by the request sender within the preset time are less concentrated and more dispersed. The larger the ratio of the number of accesses to the total number of accesses, it can be determined that the higher the access frequency, the greater the probability of a crawler. Therefore, in an embodiment of the present invention, the accuracy of the calculated stage crawler index can be ensured by calculating in a manner that the stage crawler index is positively correlated with the first usage times, positively correlated with the proportion of times, and negatively correlated with the second usage times.

[0089] It should be noted that in actual application scenarios, the P value can also be calculated in combination with other log dimensions of the access log, for example, further combined with the domain name, etc. Different log dimensions can be combined at will, and the embodiment of the present invention does not limit this.

[0090] Optionally, in the embodiment of the present invention, before determining the first usage count, the access count, and the second usage count of the access proxy corresponding to the second historical IP according to the IP access record, the following steps may be performed:

[0091] Step S81, obtaining the target request ratio corresponding to the second historical IP; the target request ratio is the ratio of historical access requests carrying the specified keyword corresponding to the second historical IP.

[0092] Specifically, the keyword may be a field used to characterize the browser used when sending the historical access request. Among them, the keyword can be extracted from the user agent field. The historical access request using the second historical IP can be determined from the IP access records within a specified preset time as the historical access request corresponding to the second historical IP. Next, the keyword is extracted from the user agent field of each historical access request corresponding to the second historical IP, and the keyword is compared with the pre-set specified keyword. If the keyword matches the specified keyword, it is determined that the historical access request carries the specified keyword. Finally, the number of historical access requests carrying the specified keyword can be counted, as well as the number of historical access requests corresponding to the second historical IP. The ratio of the number of historical access requests carrying the specified keyword to the number of historical access requests corresponding to the second historical IP is calculated to obtain the target request ratio.

[0093] Accordingly, the above operation of determining the first usage count, the access count, and the second usage count of the access proxy corresponding to the second historical IP according to the IP access record may specifically include:

[0094] Step S91, when the target request ratio is not greater than a preset ratio threshold, determine the first usage count, access count, and second usage count of the access source corresponding to the second historical IP according to the IP access record.

[0095] If the target request ratio is greater than the preset ratio threshold, it can be determined that the second historical IP does not belong to the category of crawlers that need to be identified, and the second historical IP belongs to a crawler IP that can be released, for example, a crawler belonging to a search engine. Accordingly, it can be released without counting the second historical IP. On the contrary, if the target request ratio is not greater than the preset ratio threshold, it can be determined that the second historical IP belongs to the category of crawlers that need to be identified. Therefore, the operation of determining the first number of uses, the number of visits, and the second number of uses of the access proxy corresponding to the second historical IP according to the IP access record can be performed without exceeding the preset ratio threshold, so as to calculate the P value of the second historical IP in the subsequent steps. In this way, unnecessary calculation operations can be avoided, thereby saving processing resources.

[0096] Specifically, the target request ratio is greater than the preset ratio threshold and the relevant information of the second historical IP is the specified information before it is released. Among them, the relevant information can be set according to actual needs, and the relevant information can be other information that can characterize whether the IP is a releaseable IP. For example, the relevant information can be the IP itself, and the specified information can be a crawler IP that does not belong to the crawler IP that needs to be identified. Or, the relevant information can be a user agent field, and the specified information can be a user agent field that does not belong to the crawler IP that needs to be identified. In this way, the second historical IP is released only when the target request ratio is greater than the preset ratio threshold and the relevant information is the specified information, which can avoid the crawler IP from being released by mistake to a certain extent, and then cause the problem of subsequent inability to accurately identify. Accordingly, it is also possible to perform the operation of determining the first number of uses, the number of visits, and the second number of uses of the access source corresponding to the second historical IP according to the IP access record when the target request ratio is greater than the preset ratio threshold and the relevant information of the second historical IP is not the specified information, so as to calculate the P value of the second historical IP in the subsequent steps.

[0097] In the embodiment of the present invention, the target request ratio can be obtained based on the IP reverse query module to determine whether it needs to be released. Among them, the IP reverse query module can be responsible for reverse querying the true source of the historical IP appearing in the IP log. To determine whether the historical IP is a web crawler IP that can be released, such as a crawler IP of a search engine. The IP reverse query module can call the IP reverse query interface to obtain relevant information of the second historical IP to confirm whether it needs to be released when it is found that the second historical IP uses a large number of user-agents carrying specified keywords.

[0098] The following is an explanation of a specific application scenario of an embodiment of the present invention. With the continuous development of information technology, the demand for network data collection is growing. Correspondingly, the crawler industry is growing rapidly, and the phenomenon of illegally crawling data from unauthorized platforms based on crawler technology is becoming more and more common. A large number of crawler behaviors may cause distortion of user access data, affect normal user access, and waste website server resources. In addition, the data resources provided by the website are captured in batches by illegal crawlers, which will cause losses to the website. Therefore, it is particularly important to quickly identify the crawler IP, implement anti-crawler strategies, and limit the illegal crawling of data by crawlers.

[0099] Exemplarily, the embodiments of the present invention can be implemented based on the access log parsing module, the smart contract interaction module, the IP reverse check module and the smart contract module. Among them, the smart contract module can be deployed on the blockchain, and the access log parsing module, the smart contract interaction module and the IP reverse check module can be deployed under the blockchain. The access log parsing module, the smart contract interaction module, the IP reverse check module and the smart contract module can be deployed as a system in each participating node, that is, each participating node is deployed with an access log parsing module, a smart contract interaction module, an IP reverse check module and a smart contract module. Exemplarily, it can be deployed based on cloud services. Further, in an embodiment of the present invention, a contract for a consortium chain can be pre-established. Each participating node can deploy the contract to its own server backend and listen to network requests. After successful deployment, it can be connected to the consortium chain as a new node. Among them, the participating node can refer to the aforementioned first website node.

[0100] Among them, the access log parsing module can be responsible for parsing the IP access log obtained from the application and calculating the stage crawler index. The IP reverse query module can be used to determine whether the second historical IP involved in the IP access log needs to be released. The smart contract interaction module can be used to interact with the smart contract module on the blockchain. Responsible for calling the smart contract, sending the P value obtained by the access log parsing module and the timestamp corresponding to the P value to the smart contract, and providing an API interface to facilitate the client of the website node to call to query the crawler index of the target IP. At the same time, the P value can also be synchronized to other participating nodes. Furthermore, the smart contract can be used to perform chain operations on the received P value and is responsible for calculating P' on the chain. P′ can be provided to the client through the smart contract interaction module to make decisions on the crawler strategy in the future.

[0101] For example, there are three first website nodes, all of which are deployed with blockchains, and the blockchain stores the crawler index corresponding to the first historical IP obtained based on the IP access records of the three first website nodes. Figure 2 is a schematic diagram of a scenario provided by an embodiment of the present invention, such as Figure 2 As shown, the access log parsing module, smart contract interaction module, IP reverse check module and smart contract module deployed in each first website node can interact with each other. Among them, application 1, application 2 and application 3 are the applications corresponding to the three first website nodes respectively.

[0102] In one implementation, the user can access the client's website, and the website can input the generated IP access log into the access log parsing module by data push or calling the interface. The IP contrast module can determine whether the second historical IP obtained by the analysis needs to be released. If it needs to be released, the second historical IP can be processed without being processed. If it does not need to be released, the access log parsing module can generate a stage crawler index P for the second historical IP, and call the smart contract interaction module to input the p value and timestamp of the second historical IP into the smart contract interaction module. The smart contract interaction module calls the smart contract on the blockchain, sends the received IP, the P value of the IP, and the timestamp to the smart contract module on the chain, and stores the received IP, the P value of the IP, and the timestamp in the blockchain through the smart contract module, and calculates P' based on it. Among them, these steps can be performed multiple times.

[0103] Accordingly, when the user subsequently accesses the client, the client can obtain the user IP, that is, obtain the access IP of the access request as the target IP, and call the smart contract interaction module to call the smart contract on the chain through the smart contract interaction module to obtain the P' value of the target IP. According to the set security policy threshold, if the P' value is greater than the preset index threshold, the anti-crawler strategy is triggered, otherwise it is not triggered.

[0104] In one implementation, the website's anti-crawler strategy is often a post-processing, that is, anti-crawler measures can only be implemented on the crawler IP's request after the crawler IP has visited multiple times, thus triggering the anti-crawler condition. And the anti-crawler strategy of each website is relatively independent. Therefore, it is impossible to identify the crawler IP in time.

[0105] In the embodiment of the present invention, multiple applications upload access records from different IPs together, and use smart contracts to complete the calculation of the crawler index of the IP on the blockchain, and store the IP crawler index in the blockchain, so as to realize the sharing of the IP crawler index, and use the IP access logs of multiple applications to identify the IP suspected of being a crawler. Furthermore, by using blockchain technology, a consortium chain is built, and by quantifying the IP behavior and saving the quantified crawler index in the consortium chain, the participating nodes on the chain can jointly identify the access requests suspected of being from the crawler, identify the crawler IP more quickly, and jointly maintain the crawler index and send it to different participating nodes, that is, each participating node can query the crawler index of the IP from the blockchain, so that it is convenient for each participating node to identify the crawler IP to a certain extent, and quickly implement the anti-crawler strategy.

[0106] Figure 3 : is a structural diagram of an anti-crawler processing device provided by an embodiment of the present invention. The device can be applied to a first website node. The device 20 may include:

[0107] A search module 201 is used to search for a crawler index corresponding to a target IP from the crawler index corresponding to the first historical IP in response to an access request, as a target crawler index; the first historical IP and the crawler index corresponding to the first historical IP are obtained according to IP access records of at least two second website nodes, the crawler index is used to characterize the probability that the first historical IP belongs to a crawler IP, and the target IP is the IP used by the access request;

[0108] The execution module 202 is used to execute a preset anti-crawler operation when the target crawler index is greater than a preset index threshold.

[0109] Optionally, the crawler index corresponding to the first historical IP is stored in a blockchain, and the first website node is any one of the at least two second website nodes, and the blockchain is locally deployed on any one of the first website nodes.

[0110] Optionally, the device 20 further includes:

[0111] A determination module, configured to determine, when a preset condition is met, a stage crawler index of a second historical IP within a preset time according to the IP access record of the first website node within the preset time; the stage crawler index is used to characterize the probability that the second historical IP belongs to a crawler within the preset time;

[0112] The chain module is used to chain the stage crawler index to the blockchain in the first website node, and the first generation module is used to generate a new crawler index for the second historical IP according to the stage crawler index and the historical stage crawler index of the second historical IP, and chain the new crawler index to the blockchain in the first website node; the historical stage crawler index includes the stage crawler index previously chained to the blockchain.

[0113] Optionally, the device 20 further includes:

[0114] A synchronization module is used to synchronize the stage crawler index of the second historical IP within the preset time to other first website nodes deployed with the blockchain after the determination module determines the stage crawler index of the second historical IP within the preset time, so that the other first website nodes can generate a new crawler index for the second historical IP based on the stage crawler index and the historical stage crawler index of the second historical IP, and upload the new crawler index and the received stage crawler index to the blockchain in the other first website nodes.

[0115] Optionally, the device 20 further includes:

[0116] The second generation module is used to generate a new crawler index for the third historical IP based on the stage crawler index shared by the other first website nodes and the historical stage crawler index of the third historical IP if a stage crawler index of the third historical IP shared by other first website nodes is received, and the new crawler index and the received stage crawler index are linked to the blockchain in the first website node.

[0117] Optionally, the first generating module is specifically used to:

[0118] Obtaining a historical crawler index of the second historical IP from the on-chain block of the blockchain, and obtaining a generation time of the historical crawler index as the first time;

[0119] A new crawler index is generated for the second historical IP according to the historical stage crawler index, the first time, the stage crawler index and the second time; the second time is the generation time of the stage crawler index.

[0120] Optionally, the first generating module is further specifically used for:

[0121] According to the first time and the second time, respectively determine a first weight of the historical stage crawler index and a second weight of the stage crawler index; the first weight is negatively correlated with the length of time from the first time to the current time, and the second weight is negatively correlated with the length of time from the second time to the current time;

[0122] Generate the new crawler index according to the first weight, the second weight, the historical crawler index, and the stage crawler index;

[0123] The search module 201 is specifically used to search for the latest generated crawler index corresponding to the target IP from the blockchain of the first website node.

[0124] Optionally, the determining module is specifically used to:

[0125] For the second historical IP within the preset time, determining the first usage count, the access count, and the second usage count of the access proxy corresponding to the second historical IP according to the IP access record;

[0126] The stage crawler index of the second historical IP is calculated based on the first number of usages, the number of visits, the second number of usages and the total number of visits within the preset time; the stage crawler index is positively correlated with the first number of usages and the number ratio, and negatively correlated with the second number of usages, and the number ratio is the ratio of the number of visits to the total number of visits.

[0127] Optionally, the device 20 further includes:

[0128] an acquisition module, configured to acquire a target request ratio corresponding to the second historical IP before the determination module determines the first usage count, the access count, and the second usage count of the access proxy corresponding to the second historical IP according to the IP access record; the target request ratio is a ratio of historical access requests carrying a specified keyword corresponding to the second historical IP;

[0129] The determining module is further specifically used for:

[0130] When the target request ratio is not greater than a preset ratio threshold, the first usage count, the access count, and the second usage count of the access proxy corresponding to the second historical IP are determined according to the IP access record.

[0131] In summary, the anti-crawler processing device provided by the embodiment of the present invention, in response to an access request, searches for the crawler index corresponding to the target IP from the crawler index corresponding to the first historical IP as the target crawler index; the first historical IP and the crawler index corresponding to the first historical IP are obtained according to the IP access records of at least two second website nodes, and the crawler index is used to characterize the probability that the first historical IP belongs to the crawler IP, and the target IP is the IP used for the access request. When the target crawler index is greater than the preset index threshold, the preset anti-crawler operation is performed. In this way, the crawler IP is identified by using the access records of multiple second website nodes to provide more sufficient information for crawler IP identification. The first website node uses the first historical IP obtained according to the IP access records of at least two second website nodes and its corresponding crawler index for search, and can identify the crawler IP, thereby achieving rapid identification of the crawler IP to a certain extent, and performing anti-crawler operations on it.

[0132] At the same time, compared with the method of identifying based only on the historical data of the website itself, even when the target IP visits the first website node for the first time, since the target IP may have visited a second website node before, the first website node uses the crawler index obtained according to the IP access records of at least two second website nodes for identification, which can increase the probability of finding the crawler index of the target IP to a certain extent, so that when the target IP visits for the first time, it can quickly identify and implement anti-crawler measures in time.

[0133] The present invention also provides an anti-crawler processing system, which includes at least two first website nodes, each of which is used to perform the following steps: in response to an access request, the crawler index corresponding to the target IP is searched from the crawler index corresponding to the first historical IP as the target crawler index; the first historical IP and the crawler index corresponding to the first historical IP are obtained according to the IP access records of the at least two first website nodes, and the crawler index is used to characterize the probability that the first historical IP belongs to the crawler IP, and the target IP is the IP used by the access request; when the target crawler index is greater than a preset index threshold, a preset anti-crawler operation is performed. Among them, the implementation method of each step and the technical effect that can be achieved can refer to the aforementioned related description, and will not be repeated here.

[0134] The present invention also provides an electronic device, see Figure 4 , including: a processor 901, a memory 902, and a computer program 9021 stored in the memory and executable on the processor, wherein the processor implements the anti-crawler processing method of the aforementioned embodiment when executing the program.

[0135] The present invention also provides a readable storage medium, when the instructions in the storage medium are executed by the processor of the electronic device, the electronic device can execute the anti-crawler processing method of the aforementioned embodiment.

[0136] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0137] It should be noted that the various information and data obtained in the embodiments of the present invention are all obtained with the authorization of the information / data holder.

[0138] The algorithm and display provided herein are not inherently related to any particular computer, virtual system or other device. Various general purpose systems can also be used together with the teachings based on this. According to the above description, it is obvious that the structure required for constructing such systems. In addition, the present invention is not directed to any specific programming language either. It should be understood that various programming languages ​​can be utilized to realize the content of the present invention described herein, and the description of the above specific languages ​​is for disclosing the best mode of the present invention.

[0139] In the description provided herein, a large number of specific details are described. However, it is understood that embodiments of the present invention can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this description.

[0140] Similarly, it should be understood that in order to streamline the present invention and aid in understanding one or more of the various inventive aspects, in the above description of exemplary embodiments of the present invention, the various features of the present invention are sometimes grouped together into a single embodiment, figure, or description thereof. However, this disclosed method should not be interpreted as reflecting the following intention: that the claimed invention requires more features than the features explicitly recited in each claim. More specifically, as reflected in the claims below, inventive aspects lie in less than all the features of the individual embodiments disclosed above. Therefore, the claims that follow the specific embodiment are hereby expressly incorporated into the specific embodiment, with each claim itself serving as a separate embodiment of the present invention.

[0141] Those skilled in the art will appreciate that the modules in the devices in the embodiments may be adaptively changed and arranged in one or more devices different from the embodiments. The modules or units or components in the embodiments may be combined into one module or unit or component, and in addition they may be divided into a plurality of submodules or subunits or subcomponents. Except that at least some of such features and / or processes or units are mutually exclusive, all features disclosed in this specification (including the accompanying claims, abstracts and drawings) and all processes or units of any method or device disclosed in this manner may be combined in any combination. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstracts and drawings) may be replaced by an alternative feature providing the same, equivalent or similar purpose.

[0142] The various component embodiments of the present invention may be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. It should be understood by those skilled in the art that a microprocessor or digital signal processor (DSP) may be used in practice to implement some or all of the functions of some or all of the components in the sorting device according to the present invention. The present invention may also be implemented as a device or apparatus program for executing part or all of the methods described herein. Such a program for implementing the present invention may be stored on a computer-readable medium, or may be in the form of one or more signals. Such a signal may be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

[0143] It should be noted that the above embodiments illustrate the present invention rather than limit it, and that those skilled in the art may devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference symbol between brackets shall not be construed as a limitation on the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "one" or "an" preceding an element does not exclude the presence of a plurality of such elements. The present invention may be implemented by means of hardware comprising a number of different elements and by means of a suitably programmed computer. In a unit claim enumerating a number of devices, several of these devices may be embodied by the same hardware item. The use of the words first, second, and third, etc., does not indicate any order. These words may be interpreted as names.

[0144] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0145] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

[0146] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. An anti-crawler processing method, characterized in that: Applied to a first website node, the method comprises: In response to the access request, searching for a crawler index corresponding to a target IP from the crawler index corresponding to the first historical Internet Protocol address IP as a target crawler index; the first historical IP and the crawler index corresponding to the first historical IP are obtained according to the IP access records of at least two second website nodes, the crawler index is used to characterize the probability that the first historical IP belongs to a crawler IP, and the target IP is the IP used for the access request; When the target crawler index is greater than a preset index threshold, performing a preset anti-crawler operation; The crawler index corresponding to the first historical IP is stored in a blockchain, the first website node is any one of the at least two second website nodes, and any of the first website nodes is locally deployed with the blockchain; The method further comprises: Under the condition that the preset conditions are met, according to the IP access record of the first website node within the preset time, the stage crawler index of the second historical IP within the preset time is determined; the stage crawler index is used to characterize the probability that the second historical IP belongs to a crawler within the preset time; The stage crawler index is chained to the blockchain in the first website node, and, based on the stage crawler index and the historical stage crawler index of the second historical IP, a new crawler index is generated for the second historical IP, and the new crawler index is chained to the blockchain in the first website node; the historical stage crawler index includes the stage crawler index previously chained to the blockchain.

2. The method according to claim 1, characterized in that After determining the stage crawler index of the second historical IP within the preset time, the method further includes: The stage crawler index is synchronized to other first website nodes where the blockchain is deployed, so that the other first website nodes can generate a new crawler index for the second historical IP based on the stage crawler index and the historical stage crawler index of the second historical IP, and the new crawler index and the received stage crawler index are linked to the blockchain in the other first website nodes.

3. The method according to claim 1, characterized in that The method further comprises: If a stage crawler index of a third historical IP shared by other first website nodes is received, a new crawler index is generated for the third historical IP based on the stage crawler index shared by the other first website nodes and the historical stage crawler index of the third historical IP, and the new crawler index and the received stage crawler index are uploaded to the blockchain in the first website node.

4. The method according to any one of claims 1 to 3, characterized in that: Generating a new crawler index for the second historical IP according to the crawler index of the stage and the crawler index of the second historical IP in the historical stage includes: Obtaining a historical crawler index of the second historical IP from the on-chain block of the blockchain, and obtaining a generation time of the historical crawler index as the first time; A new crawler index is generated for the second historical IP according to the historical stage crawler index, the first time, the stage crawler index and the second time; the second time is the generation time of the stage crawler index.

5. The method according to claim 4, characterized in that Generating a new crawler index for the second historical IP according to the historical stage crawler index, the first time, the stage crawler index and the second time includes: According to the first time and the second time, respectively determine a first weight of the historical stage crawler index and a second weight of the stage crawler index; the first weight is negatively correlated with the length of time from the first time to the current time, and the second weight is negatively correlated with the length of time from the second time to the current time; Generate the new crawler index according to the first weight, the second weight, the historical crawler index, and the stage crawler index; The searching for the crawler index corresponding to the target IP from the crawler index corresponding to the first historical Internet Protocol address IP includes: searching for the latest generated crawler index corresponding to the target IP from the blockchain of the first website node.

6. The method according to any one of claims 1 to 3, characterized in that: The step of determining the stage crawler index of the second historical IP within the preset time according to the IP access record of the first website node within the preset time includes: For the second historical IP within the preset time, determining the first usage count, the access count, and the second usage count of the access proxy corresponding to the second historical IP according to the IP access record; The stage crawler index of the second historical IP is calculated based on the first number of usages, the number of visits, the second number of usages and the total number of visits within the preset time; the stage crawler index is positively correlated with the first number of usages and the number ratio, and negatively correlated with the second number of usages, and the number ratio is the ratio of the number of visits to the total number of visits.

7. The method according to claim 6, characterized in that Before determining the first usage count, the access count, and the second usage count of the access proxy corresponding to the second historical IP according to the IP access record, the method further includes: Obtain the target request ratio corresponding to the second historical IP; the target request ratio is the ratio of historical access requests carrying a specified keyword corresponding to the second historical IP; The determining, according to the IP access record, a first usage count, an access count, and a second usage count of an access proxy corresponding to the second historical IP, comprises: When the target request ratio is not greater than a preset ratio threshold, the first usage count, the access count, and the second usage count of the access proxy corresponding to the second historical IP are determined according to the IP access record.

8. An anti-crawler processing device, characterized in that: Applied to a first website node, the device comprises: a search module, configured to search for a crawler index corresponding to a target IP from the crawler index corresponding to the first historical Internet Protocol address IP in response to an access request, as a target crawler index; the first historical IP and the crawler index corresponding to the first historical IP are obtained according to IP access records of at least two second website nodes, the crawler index is used to characterize the probability that the first historical IP belongs to a crawler IP, and the target IP is the IP used by the access request; An execution module, configured to execute a preset anti-crawler operation when the target crawler index is greater than a preset index threshold; The crawler index corresponding to the first historical IP is stored in a blockchain, the first website node is any one of the at least two second website nodes, and any of the first website nodes is locally deployed with the blockchain; The device is also used for: Under the condition that the preset conditions are met, according to the IP access record of the first website node within the preset time, the stage crawler index of the second historical IP within the preset time is determined; the stage crawler index is used to characterize the probability that the second historical IP belongs to a crawler within the preset time; The stage crawler index is chained to the blockchain in the first website node, and, based on the stage crawler index and the historical stage crawler index of the second historical IP, a new crawler index is generated for the second historical IP, and the new crawler index is chained to the blockchain in the first website node; the historical stage crawler index includes the stage crawler index previously chained to the blockchain.

9. An anti-crawler processing system, characterized in that: The system includes at least two first website nodes, each of which is used to perform the following steps: In response to the access request, searching the crawler index corresponding to the target IP from the crawler index corresponding to the first historical IP as the target crawler index; the first historical IP and the crawler index corresponding to the first historical IP are obtained according to the IP access records of the at least two first website nodes, the crawler index is used to characterize the probability that the first historical IP belongs to the crawler IP, and the target IP is the IP used by the access request; When the target crawler index is greater than a preset index threshold, performing a preset anti-crawler operation; The crawler index corresponding to the first historical IP is stored in a blockchain, the first website node is any one of the at least two second website nodes, and any of the first website nodes is locally deployed with the blockchain; The first website node is further used to determine, if a preset condition is met, a stage crawler index of a second historical IP within a preset time according to the IP access record of the first website node within the preset time; the stage crawler index is used to characterize the probability that the second historical IP belongs to a crawler within the preset time; The stage crawler index is chained to the blockchain in the first website node, and, based on the stage crawler index and the historical stage crawler index of the second historical IP, a new crawler index is generated for the second historical IP, and the new crawler index is chained to the blockchain in the first website node; the historical stage crawler index includes the stage crawler index previously chained to the blockchain.

10. An electronic device, characterized in that: include: A processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method according to any one of claims 1 to 7 when executing the program.

11. A readable storage medium, characterized in that: When the instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute any one of the methods described in claims 1-7.