Anti-crawler method and device based on blockchain and electronic equipment

Abstract

Embodiments of the present application provide a blockchain-based anti-crawler method and device and electronic equipment, relating to the technical field of computer. The method comprises: saving first target access information corresponding to a target access request received and sent by a second device to a distributed ledger of each blockchain node of a blockchain network; analyzing whether the target access request is a crawler access request according to a second target access information set corresponding to the first target access information in the distributed ledger, the second target access information set comprising the first target access information and the corresponding target access initiator being an access initiator corresponding to the first target access information; and processing the target access request according to a preset anti-crawling measure when the target access request is a crawler access request. In this way, by applying blockchain technology to anti-crawling, the reliability and accuracy of crawler detection can be improved, and the safety and privacy of the website can be protected by preventing malicious crawler access.

Description

Technical Field

[0001] This application relates to the field of computer technology, and more specifically, to a blockchain-based anti-scraping method, apparatus, and electronic device. Background Technology

[0002] In the current network environment, web crawling technology is widely used in fields such as data collection and search engine optimization. However, in some cases, malicious web crawling activities may lead to security problems, information leaks, or a decline in service quality. Therefore, how to effectively deal with web crawling issues has become a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0003] This application provides a blockchain-based anti-crawler method, device, electronic device, and readable storage medium. By utilizing the distributed storage and consensus mechanism of blockchain, it enhances the security and immutability of data, thereby improving the reliability and accuracy of crawler detection. In turn, it prevents malicious crawlers from accessing the website when crawlers are detected, thus protecting the website's security and privacy.

[0004] The embodiments of this application can be implemented as follows:

[0005] In a first aspect, embodiments of this application provide a blockchain-based anti-scraping method, the method comprising:

[0006] The first target access information corresponding to the target access request sent by the second device is saved to the distributed ledger of each blockchain node in the blockchain network, wherein the distributed ledger includes the access information corresponding to the second device.

[0007] Based on the second target access information set in the distributed ledger that corresponds to the first target access information, analyze whether the target access request is a crawler access request, wherein the second target access information set includes the first target access information, and the corresponding target access initiator is the access initiator corresponding to the first target access information;

[0008] When the target access request is the crawler access request, the target access request is processed according to preset anti-crawling measures.

[0009] Secondly, embodiments of this application provide a blockchain-based anti-scraping device, the device comprising:

[0010] The storage module is used to store the first target access information corresponding to the target access request sent by the second device to the distributed ledger of each blockchain node in the blockchain network, wherein the distributed ledger includes the access information corresponding to the second device.

[0011] The analysis module is used to analyze whether the target access request is a crawler access request based on the second target access information set in the distributed ledger that corresponds to the first target access information. The second target access information set includes the first target access information, and the corresponding target access initiator is the access initiator corresponding to the first target access information.

[0012] The processing module is used to process the target access request according to preset anti-crawling measures when the target access request is the crawler access request.

[0013] Thirdly, embodiments of this application provide an electronic device, including a processor and a memory, wherein the memory stores machine-executable instructions that can be executed by the processor, and the processor can execute the machine-executable instructions to implement the blockchain-based anti-crawler method described in the foregoing embodiments.

[0014] Fourthly, embodiments of this application provide a readable storage medium storing a computer program thereon, which, when executed by a processor, implements the blockchain-based anti-crawler method as described in the foregoing embodiments.

[0015] The blockchain-based anti-crawler method, apparatus, electronic device, and readable storage medium provided in this application, upon receiving a target access request from a second device, determines first access information based on the target access request and stores the first access information on the distributed ledger of each blockchain node in the blockchain network. The distributed ledger includes access information corresponding to the second device. Then, based on a second target access information set corresponding to the first target access information in the distributed ledger, it analyzes whether the target access request is a crawler access request. If so, it processes the target access request according to preset anti-crawler measures. The second target access information set includes the first target access information, and the target access initiator corresponding to the second target access information set is the access initiator corresponding to the first target access information. Thus, by utilizing the distributed storage and consensus mechanism of blockchain, data security and immutability can be enhanced, thereby improving the reliability and accuracy of crawler detection; preventing malicious crawler access when crawlers are detected can protect website security and privacy. Attached Figure Description

[0016] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 A block diagram illustrating an electronic device provided in an embodiment of this application;

[0018] Figure 2 A flowchart illustrating the blockchain-based anti-scraping method provided in this application embodiment;

[0019] Figure 3 for Figure 2 A flowchart illustrating the sub-steps included in step S120;

[0020] Figure 4 for Figure 3 A flowchart illustrating the sub-steps included in the neutron step S122;

[0021] Figure 5 A block diagram illustrating a blockchain-based anti-crawler device provided in an embodiment of this application.

[0022] Icons: 100 - Electronic device; 110 - Memory; 120 - Processor; 130 - Communication unit; 200 - Blockchain-based anti-crawler device; 210 - Storage module; 220 - Analysis module; 230 - Processing module. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0024] Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0025] It should be noted that relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0026] Currently, several anti-scraping technologies exist, such as CAPTCHAs, IP blocking, and User-Agent detection. These solutions typically rely on mechanisms like human-machine identification, IP address restrictions, and request header checks to identify and block web crawler access. However, CAPTCHAs are user-unfriendly and can lead to access difficulties; furthermore, these solutions lack credibility and immutability, making it difficult to ensure the reliability of crawler detection results. For example, these solutions first extract and save target information from the information to be analyzed, and then perform crawler detection based on that target information. If the target information in the crawler detection device is tampered with during this process, it can lead to inaccurate and unreliable crawler detection results.

[0027] In response to the above situation, this application provides a blockchain-based anti-crawler method, device, electronic device, and readable storage medium. By applying blockchain technology to anti-crawler work, the reliability and accuracy of crawler detection can be improved, thereby protecting website security and privacy by preventing malicious crawlers from accessing the site.

[0028] The following detailed description of some embodiments of this application is provided in conjunction with the accompanying drawings. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0029] Please refer to Figure 1 , Figure 1 This is a block diagram of an electronic device 100 provided in an embodiment of this application. The electronic device 100 may be, but is not limited to, a computer, a server, etc. The electronic device 100 includes a memory 110, a processor 120, and a communication unit 130. The memory 110, processor 120, and communication unit 130 are electrically connected to each other directly or indirectly to achieve data transmission or interaction. For example, these components can be electrically connected to each other through one or more communication buses or signal lines.

[0030] The memory 110 is used to store programs or data. The memory 110 may be, but is not limited to, random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), etc.

[0031] The processor 120 is used to read / write data or programs stored in the memory 110 and execute corresponding functions. For example, the memory 110 stores a blockchain-based anti-crawler device 200, which includes at least one software functional module that can be stored in the memory 110 in the form of software or firmware. The processor 120 executes various functional applications and data processing by running the software programs and modules stored in the memory 110, such as the blockchain-based anti-crawler device 200 in this embodiment, thereby implementing the blockchain-based anti-crawler method in this embodiment.

[0032] The communication unit 130 is used to establish a communication connection between the electronic device 100 and other communication terminals through the network, and to send and receive data through the network.

[0033] It should be understood that, Figure 1 The structure shown is only a schematic diagram of the electronic device 100. The electronic device 100 may also include components that are larger than... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown. Figure 1 The components shown can be implemented using hardware, software, or a combination thereof.

[0034] Please refer to Figure 2 , Figure 2 This is a flowchart illustrating a blockchain-based anti-scraping method provided in an embodiment of this application. The method can be applied to a first device, which can be the aforementioned electronic device 100. The specific flow of the blockchain-based anti-scraping method is described in detail below. In this embodiment, the method may include steps S110 to S130.

[0035] Step S110: Save the first target access information corresponding to the target access request sent by the second device to the distributed ledger of each blockchain node in the blockchain network.

[0036] In this embodiment, the first device is the device that receives the access request, and the second device is the device that sends the access request; the specific configuration can be tailored to actual needs. The first device receives the target access request sent by the second device, then extracts information from the target access request to obtain first target access information corresponding to the target access request. This first target access information is then sent to the blockchain network for storage, ensuring that the first target access information is included in the distributed ledger of each blockchain node in the blockchain network. Each blockchain node maintains the distributed ledger based on a consensus mechanism.

[0037] The target access request refers to the access request currently received by the first device. When the first device receives an access request from any second device, it can obtain the corresponding access information based on the request and store it on the distributed ledger of each blockchain node in the blockchain network. Thus, the distributed ledger can include the access information corresponding to each second device.

[0038] Optionally, when the access request is for accessing a protected resource, the access information corresponding to the access request is saved in the distributed ledger.

[0039] Step S120: Analyze whether the target access request is a crawler access request based on the second target access information set in the distributed ledger that corresponds to the first target access information.

[0040] When analyzing and determining whether a target access request is a web crawler request, the first device can obtain a second target access information set corresponding to the first target access information from the distributed ledger. The second target access information set includes the first target access information, and the access initiator corresponding to each entry in the second target access information set is the same. That is, when the access initiation text corresponding to the second target access information set is named the target access initiator, this target initiator is the access initiator corresponding to the first target access information. Thus, when historical access information is stored in the distributed ledger, it can be used to make the determination. The second target access information set can include only one access message or multiple messages, depending on the actual needs.

[0041] The initiator of the access can refer to a device, a user, or an IP address, depending on the specific requirements. That is, the access information corresponding to an initiator can refer to access information from access requests sent by the same device, access information from access requests sent using the same user identifier, or access information from access requests sent using the same IP address, where the user identifier is unique.

[0042] Having obtained the second target access information set, corresponding features can be extracted from it for crawler detection based on the chosen judgment method. For example, if the judgment method requires IP address-based detection, the source IP address corresponding to the request can be extracted from the second target access information set, and then crawler detection can be performed based on that source IP address. It is understood that the above method of crawler detection based on IP address is merely an example, and other methods can also be used for crawler detection.

[0043] In this way, the consensus mechanism of blockchain can be used to ensure the accuracy and credibility of web crawler detection results through verification and joint processing by multiple nodes.

[0044] Step S130: When the target access request is the crawler access request, the target access request is processed according to preset anti-crawling measures.

[0045] If it is determined that the target access request is not a web crawler request, the request can be processed normally to allow the second device to obtain the corresponding request. If it is determined that the target access request is a web crawler request, preset anti-crawling measures can be used to prevent malicious web crawlers from accessing the site and protect website security and privacy. The specific preset anti-crawling measures can be set according to actual circumstances and are not specifically limited here.

[0046] This application's embodiments enhance data security and immutability by utilizing blockchain's distributed storage and consensus mechanism, thereby improving the reliability and accuracy of crawler detection; when a crawler is detected, blocking malicious crawler access can protect website security and privacy.

[0047] Optionally, in this embodiment, upon receiving the target access request, the first device can obtain access time, user identifier, request method (e.g., POST, GET), request information, etc., from the target access request to form the first target access information. The request information can be used to determine the data scraping method. Optionally, the request information may include any one of: request headers, target resource description information, data acquisition method, etc., wherein the target resource description information describes the resource the access request expects to obtain. It is understood that each piece of access information includes access time, user identifier, request information, etc. After obtaining the first target access information, the first device can send the first target access information to the blockchain network for storage.

[0048] Optionally, in this embodiment, the blockchain network may include a first blockchain node and at least one second blockchain node connected via communication. The first blockchain node is connected to the first device. The first device can send the first target access information to the first blockchain node. The first blockchain node can use the blockchain's broadcast mechanism, specific transactions or smart contracts, or consensus algorithms to enable other nodes in the second blockchain to obtain the first target access information. These mechanisms ensure information synchronization and consistency among nodes in the blockchain network, thereby achieving the effectiveness and accuracy of multi-node verification. In this way, the first target access information can be saved on the blockchain's distributed ledger.

[0049] Storing access information in the blockchain's distributed ledger provides data security, trustworthiness, historical records, traceability, and decentralized distributed storage. Data security refers to the fact that the blockchain uses encryption algorithms and decentralization to ensure the security of access information; each block contains the hash value of the previous block, forming an immutable data chain that protects data integrity. Data trustworthiness refers to the fact that the blockchain's consensus mechanism requires multiple nodes to verify and jointly maintain data, ensuring consistency and trustworthiness. Therefore, recording access information in the blockchain increases data trustworthiness and reduces the possibility of data tampering. Historical records and traceability refer to the ability to preserve and trace historical records by recording access information in the blockchain. This allows for reference during web scraping detection and anti-scraping processes, aiding in judgment and decision-making and improving detection accuracy. Decentralized and distributed storage refers to the fact that the blockchain uses a decentralized storage method, distributing access information across multiple nodes. This distributed storage method improves the system's robustness and resistance to attacks, preventing single points of failure and data loss.

[0050] Please refer to Figure 3 , Figure 3 for Figure 2 A flowchart illustrating the sub-steps included in step S120. In this embodiment, step S120 may include sub-steps S121 to S123.

[0051] Sub-step S121: Obtain the second target access information set from each blockchain node.

[0052] In this embodiment, for the distributed ledger in each blockchain node, the first device can obtain the second target access information set from the distributed ledger, thereby obtaining multiple second target access information sets. For example, if the blockchain network includes 3 blockchain nodes, then one second target access information set is obtained from each of the 3 blockchain nodes, resulting in a total of 3 second target access information sets.

[0053] Sub-step S122: For each of the second target access information sets, perform an initial judgment on whether the target access request is the crawler access request, and obtain an initial judgment result.

[0054] In this embodiment, the first device can use any crawler identification method to perform an initial judgment on each of the second target access information sets, thereby obtaining the initial judgment result corresponding to each second target access information set. The specific crawler identification method used can be set according to actual needs and is not specifically limited here.

[0055] As one possible implementation method, it can be achieved through Figure 4 The initial judgment result is obtained as shown. Please refer to... Figure 4 , Figure 4 for Figure 3 A flowchart illustrating the sub-steps included in neutron step S122. In this embodiment, sub-step S122 may include sub-steps S1221 to S1223.

[0056] Sub-step S1221: Analyze the behavior pattern to be analyzed based on the second target access information set.

[0057] Sub-step S1222: The behavior pattern to be analyzed is compared with the stored crawler behavior patterns to obtain the similarity between the behavior pattern to be analyzed and the stored crawler behavior patterns.

[0058] Sub-step S1223: Obtain the initial judgment result based on the obtained similarity.

[0059] In this embodiment, the second target access information set may include historical access information corresponding to the target access initiator of the second device and the first target access information. Each access information entry includes access time, user identifier, request method, request information, etc. The request information may include any one of the following: target resource description information, data acquisition method, etc. The data crawling method includes at least one of data crawling path, data acquisition method, and request header. The feature types of the behavior pattern to be analyzed and the crawler behavior pattern may both include user identifier, request frequency, data crawling method, etc. Thus, analysis can be performed based on user identifier, request frequency, data crawling method, etc.

[0060] Request frequency analysis refers to analyzing the frequency and time intervals of access requests to detect excessively frequent or regular request patterns. Normal users typically have a certain request interval and randomness, while web crawlers may scrape data in a high-frequency, continuous manner.

[0061] Request header analysis refers to examining request header information in access requests, such as the User-Agent header, to identify whether common crawler user agent identifiers are used. Regular browsers typically use User-Agent with a specific format, while crawlers may use custom or common crawler agent identifiers.

[0062] Data crawling path analysis refers to analyzing the path and order of access requests to determine whether they follow the normal page navigation pattern of users. A crawler may directly access a specific target page or access pages sequentially according to a specific page structure.

[0063] Data acquisition method analysis refers to examining how data is acquired, such as whether it's obtained through API interfaces, by parsing webpage HTML, or by simulating user interactions. Normal users typically use browsers to access and interact with web pages, while web crawlers may use programmatic methods to acquire data.

[0064] Given the second target access information set, corresponding features can be extracted based on the feature types of the second target access information set and the crawler behavior pattern, thereby obtaining the behavior pattern to be analyzed corresponding to the second target access information set. Then, for each crawler behavior pattern, the similarity between the crawler behavior pattern and the behavior pattern to be analyzed is calculated. Optionally, for a crawler behavior pattern and the behavior pattern to be analyzed, the feature similarity between each feature can be calculated, and then the similarity between the crawler behavior pattern and the behavior pattern to be analyzed can be obtained based on the obtained feature similarity between various features.

[0065] Optionally, various similarity measurement methods can be used to calculate behavioral pattern similarity. These include rule-based matching and machine learning algorithms. In rule-based matching, the crawler's behavioral patterns can include matching rules corresponding to various features. If any feature in the pattern to be analyzed satisfies the corresponding matching rule, the feature similarity to the feature corresponding to that matching rule can be considered to be a preset value greater than 0. If not, the feature similarity to the feature corresponding to that matching rule can be considered to be 0. The behavioral pattern similarity can then be obtained by summing the feature similarities between various features.

[0066] After obtaining the similarity between the behavior pattern to be analyzed and a crawler behavior pattern, it can be determined whether the similarity is greater than a preset similarity. If it is greater, the initial judgment result is that the target access request is a crawler access request; if it is greater, the initial judgment result is that the target access request is not a crawler access request. This process is repeated to obtain the initial judgment result for each second access information set.

[0067] Sub-step S122: Obtain the analysis result based on the initial judgment result corresponding to each of the second target access information sets.

[0068] In this embodiment, given the initial judgment results corresponding to each second access information set, the analysis result can be determined based on multiple initial judgment results using any method. The analysis result indicates whether the target access request is a crawler access request or not.

[0069] For example, methods such as majority decision, weighted evaluation, threshold judgment, and credibility assessment can be used. The majority decision method means that if a majority of the initial judgments indicate that the target access request is not a crawler access request, then the analysis result is that the target access request is not a crawler access request; conversely, if a majority of the initial judgments indicate that the target access request is a crawler access request, then the analysis result is determined to be that the target access request is a crawler access request. The weighted evaluation method means that each blockchain node has a pre-set weight, and the analysis result is determined based on the weight of each blockchain node corresponding to each initial judgment result. The initial judgment result of the blockchain node with a larger weight has a greater impact on the analysis result. The threshold judgment method means that the analysis result is determined based on the number or proportion of initial judgment results indicating that the target access request is not a crawler access request. For example, if the number or proportion received is greater than a set threshold, then the analysis result is determined to be that the target access request is not a crawler access request; conversely, if the number or proportion received is less than a set threshold, then the analysis result is determined to be that the target access request is a crawler access request. The credibility assessment method refers to determining the credibility of each blockchain node based on its historical records. For example, if a blockchain node has experienced a failure, its credibility is considered low. The analysis results are then determined based on the credibility of each blockchain node corresponding to each initial assessment result. This approach increases the reliability and accuracy of the analysis results by building upon multi-node verification.

[0070] When the analysis results indicate that the target access request is a crawler access request, an error message can be returned for the target access request, or a redirection message can be returned to redirect to other pages, etc., to prevent crawling.

[0071] As one possible implementation, after obtaining the analysis results, these results can also be saved to the distributed ledger of each blockchain node in the blockchain network. The first device can adjust (i.e., optimize) and / or add crawler behavior patterns based on the analysis results stored in the blockchain network and the corresponding second target access information set. The specific timing for updating the crawler behavior patterns can be set according to actual needs and is not specifically limited here; for example, adjustments can be made at relatively long intervals.

[0072] In this embodiment, the distributed storage and consensus mechanism based on blockchain enhances data security and immutability, thereby improving the reliability and accuracy of web crawler detection and reducing false positives and false negatives. Furthermore, it blocks malicious web crawler access when detected, thus protecting website security and privacy. This method also avoids cumbersome human-machine verification operations, improving user experience. Thus, this embodiment utilizes the decentralized, immutable, and trustworthy characteristics of blockchain to provide a secure and reliable web crawler detection and defense mechanism.

[0073] To perform the corresponding steps in the above embodiments and various possible methods, an implementation of a blockchain-based anti-crawler device 200 is given below. Optionally, the blockchain-based anti-crawler device 200 can adopt the above-described... Figure 1 The device structure of the electronic device 100 shown. Further, please refer to... Figure 5 , Figure 5 This is a block diagram illustrating a blockchain-based anti-crawler device 200 provided in this embodiment. It should be noted that the basic principle and technical effects of the blockchain-based anti-crawler device 200 provided in this embodiment are the same as those in the above embodiments. For the sake of brevity, any parts not mentioned in this embodiment can be referred to the corresponding content in the above embodiments. The blockchain-based anti-crawler device 200 may include: a storage module 210, an analysis module 220, and a processing module 230.

[0074] The storage module 210 is used to store the first target access information corresponding to the target access request sent by the second device to the distributed ledger of each blockchain node in the blockchain network. The distributed ledger includes the access information corresponding to the second device.

[0075] The analysis module 220 is used to analyze whether the target access request is a crawler access request based on a second target access information set in the distributed ledger that corresponds to the first target access information. The second target access information set includes the first target access information, and the corresponding target access initiator is the access initiator corresponding to the first target access information.

[0076] The processing module 230 is used to process the target access request according to preset anti-crawling measures when the target access request is the crawler access request.

[0077] Optionally, the above modules can be stored in the form of software or firmware. Figure 1 The memory 110 shown is either stored in or embedded in the operating system (OS) of the electronic device 100, and can be used by... Figure 1 The processor 120 executes the program. Meanwhile, the data and program code required to execute the above modules can be stored in the memory 110.

[0078] This application also provides a readable storage medium storing a computer program thereon, which, when executed by a processor, implements the blockchain-based anti-crawler method described above.

[0079] In summary, this application provides a blockchain-based anti-crawler method, apparatus, electronic device, and readable storage medium. Upon receiving a target access request from a second device, first access information is determined based on the target access request, and this first access information is stored on the distributed ledger of each blockchain node in the blockchain network. The distributed ledger includes access information corresponding to the second device. Then, based on a second target access information set corresponding to the first target access information in the distributed ledger, it is analyzed whether the target access request is a crawler access request. If so, the target access request is processed according to preset anti-crawler measures. The second target access information set includes the first target access information, and the target access initiator corresponding to the second target access information set is the same as the access initiator corresponding to the first target access information. Thus, by utilizing the distributed storage and consensus mechanism of blockchain, data security and immutability can be enhanced, thereby improving the reliability and accuracy of crawler detection. Preventing malicious crawler access when crawlers are detected protects website security and privacy.

[0080] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can also be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0081] In addition, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0082] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0083] The above description is merely an optional embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A blockchain-based anti-scraping method, characterized in that, Applied to a first device, the method includes: The first target access information corresponding to the target access request sent by the second device is saved to the distributed ledger of each blockchain node in the blockchain network, wherein the distributed ledger includes the access information corresponding to the second device. Based on the second target access information set in the distributed ledger that corresponds to the first target access information, it is analyzed whether the target access request is a crawler access request. The second target access information set includes historical access information corresponding to the target access initiator and the first target access information, and the corresponding target access initiator is the access initiator corresponding to the first target access information. When the target access request is the crawler access request, the target access request is processed according to preset anti-crawling measures; The step of analyzing whether the target access request is a web crawler access request based on the second target access information set corresponding to the first target access information in the distributed ledger includes: obtaining the second target access information set from each blockchain node; performing an initial judgment on whether the target access request is a web crawler access request for each second target access information set, and obtaining a corresponding initial judgment result; obtaining an analysis result based on the initial judgment result corresponding to each second target access information set, wherein the analysis result is used to indicate whether the target access request is a web crawler access request or whether the target access request is not a web crawler access request; The step of performing an initial judgment on whether the target access request is the crawler access request for each of the second target access information sets, and obtaining the corresponding initial judgment result, includes: analyzing the behavior pattern to be analyzed based on the second target access information set; comparing the behavior pattern to be analyzed with each of the stored crawler behavior patterns to obtain the similarity between the behavior pattern to be analyzed and each of the stored crawler behavior patterns; and obtaining the initial judgment result based on the obtained similarity.

2. The method according to claim 1, characterized in that, Each access information entry includes access time, user ID, and request information. The feature types of the behavior pattern to be analyzed and the crawler behavior pattern both include user ID, request frequency, and data crawling method. The data crawling method is determined based on the request information. The step of comparing the behavior pattern to be analyzed with each stored crawler behavior pattern to obtain the similarity between the behavior pattern to be analyzed and each stored crawler behavior pattern includes: For the behavior pattern to be analyzed and a crawler behavior pattern, calculate the feature similarity between the same feature; Based on the feature similarity between the various features obtained, the similarity between the behavior pattern to be analyzed and the crawler behavior pattern is obtained.

3. The method according to claim 2, characterized in that, The data crawling method includes at least one of the following: data crawling path, data acquisition method, and request header.

4. The method according to claim 1, characterized in that, The method further includes: Based on the obtained analysis results and the corresponding second target access information set, adjust and / or add crawler behavior patterns to the stored crawler behavior patterns.

5. The method according to any one of claims 1-4, characterized in that, When the target access request is a crawler access request, the process of handling the target access request according to preset anti-crawling measures includes: The system returns an error message or a redirection message to redirect to another page in response to the target access request.

6. A blockchain-based anti-crawler device, characterized in that, The device includes: The storage module is used to store the first target access information corresponding to the target access request sent by the second device to the distributed ledger of each blockchain node in the blockchain network, wherein the distributed ledger includes the access information corresponding to the second device. The analysis module is used to analyze whether a target access request is a web crawler access request based on a second target access information set corresponding to the first target access information in the distributed ledger. The second target access information set includes historical access information corresponding to the target access initiator and the first target access information, and the corresponding target access initiator is the access initiator corresponding to the first target access information. The step of analyzing whether a target access request is a web crawler access request based on the second target access information set corresponding to the first target access information in the distributed ledger includes: obtaining the second target access information set from each blockchain node; and, for each second target access information set, determining whether the target access request is a web crawler access request. The initial judgment is made to obtain the corresponding initial judgment result; based on the initial judgment result corresponding to each of the second target access information sets, an analysis result is obtained, wherein the analysis result is used to indicate whether the target access request is a crawler access request or the target access request is not a crawler access request; the initial judgment of whether the target access request is a crawler access request for each of the second target access information sets, and obtaining the corresponding initial judgment result, includes: analyzing the behavior pattern to be analyzed based on the second target access information set; comparing the behavior pattern to be analyzed with each stored crawler behavior pattern to obtain the similarity between the behavior pattern to be analyzed and each stored crawler behavior pattern; and obtaining the initial judgment result based on the obtained similarity. The processing module is used to process the target access request according to preset anti-crawling measures when the target access request is the crawler access request.

7. An electronic device, characterized in that, It includes a processor and a memory, the memory storing machine-executable instructions that can be executed by the processor, the processor executing the machine-executable instructions to implement the blockchain-based anti-crawler method according to any one of claims 1-5.

8. A readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the blockchain-based anti-crawler method as described in any one of claims 1-5.

Images