Malicious crawler recognition system based on block chain

Abstract

The invention belongs to the field of computers, and particularly relates to a malicious crawler recognition system based on a block chain, comprising a malicious crawler detection module configured to capture network traffic and request information and extract features for training a machine learning model to recognize normal traffic and crawler behaviors; the data recording module is designed to be used for collecting output results of the malicious crawler detection module and recording the data in a standard format; the intelligent contract module comprises an intelligent contract used for data storage, query and record updating, and the contract is compiled and deployed on a block chain network; the block chain module selects and configures a block chain platform, designs and realizes a smart contract, processes the combination of on-chain data storage and off-chain data storage, and monitors the operation state of a block chain network; and the data synchronization and sharing module is configured to realize real-time synchronization and regular synchronization tasks when new data or data change is detected, and provides data access service through an API (Application Program Interface). And the malicious crawler identification accuracy is improved.

Description

Technical Field

[0001] The present invention belongs to the field of computers, and in particular relates to a malicious crawler identification system based on blockchain. Background Art

[0002] With the rapid development of Internet technology, web crawler technology has been widely used in data collection, search engine optimization and other fields. However, the existence of malicious crawlers has brought many troubles to website operators, such as data leakage, excessive server load, illegal content copying and other problems.

[0003] In the current context, the main malicious crawler detection technologies are: (1) Traffic analysis tools: Google Analytics: Mainly used to analyze user traffic, may not be sophisticated enough to identify abnormal behavior of crawlers. Web Server Logs: Analyzing logs requires processing a large amount of data, which may result in high latency and resource consumption. Log files may also be difficult to parse and analyze in real time. (2) Real-time traffic monitoring: Splunk: Although powerful, it may require high resource consumption and cost, and may be too complex for small websites. ELK Stack: It is relatively complex to deploy and maintain, requires specialized technical support, and may incur high storage and computing costs. (3) IP blacklist and reputation scoring: Cloudflare: Relies on known blacklist and reputation scoring systems, which may not effectively identify new or dynamically changing malicious IPs. MaxMind: IP reputation scoring may have misjudgments, resulting in legitimate users being mistakenly labeled as malicious users.

[0004] Traditional malicious crawler identification methods mostly rely on log analysis, behavior pattern matching and other means, but these methods often fail to achieve ideal identification effects when faced with complex and changeable malicious crawler behaviors. Summary of the invention

[0005] Traditional malicious crawler identification methods mostly rely on log analysis, behavior pattern matching and other means, but these methods are often difficult to achieve ideal identification effects when faced with complex and changeable malicious crawler behaviors. The present invention provides a malicious crawler identification system based on blockchain.

[0006] The technical solution of the present invention provides a malicious crawler identification system based on blockchain, including a malicious crawler detection module, a data recording module, a smart contract module, a blockchain module, and a data synchronization and sharing module; The malicious crawler detection module is configured to capture network traffic and request information, define detection rules, implement a rule engine to mark potential malicious requests, and extract features for training machine learning models to identify normal traffic and crawler behavior; The data recording module is designed to collect the output results of the malicious crawler detection module and record these data in a standard format; The smart contract module includes smart contracts for data storage, query and record update. The contracts are compiled and deployed on the blockchain network and support interaction with the blockchain through contract methods. The blockchain module selects and configures the blockchain platform, designs and implements smart contracts, handles the combination of on-chain data and off-chain data storage, and monitors the operating status of the blockchain network; The data synchronization and sharing module is configured to implement real-time synchronization and periodic synchronization tasks when new data or data changes are detected, and provides data access services through the API interface.

[0007] The malicious crawler detection module captures network traffic and request information, defines detection rules, and combines machine learning models to identify normal traffic and crawler behavior, which can greatly improve the accuracy of malicious crawler identification. Compared with traditional identification methods, the system can more accurately identify malicious crawler behavior and reduce false positives and false negatives.

[0008] The system adopts a modular design, and each module is relatively independent and can be flexibly expanded and upgraded according to actual needs. This helps to adapt to future changes in the network environment and the development of malicious crawler behavior, and maintain the advancement and practicality of the system.

[0009] As a further limitation of the technical solution of the present invention, the system also includes: an incentive mechanism module, which designs and deploys smart contracts, sets incentive rules, and encourages users to participate in malicious crawler detection. Specifically, the incentive mechanism module automatically distributes rewards according to the contributions of participants, automatically calculates the reward amount through smart contracts, sets incentive rules in smart contracts, stores incentive records on the blockchain, collects feedback data from participants, and adjusts incentive rules based on feedback information.

[0010] The incentive mechanism module is an important part of the malicious crawler identification system based on blockchain. Its main goal is to encourage users to participate in detection, provide data and maintain the operation of the system through incentives. This module uses the blockchain's smart contract to design and manage incentive mechanisms to ensure the system's enthusiasm and sustainability.

[0011] As a further limitation of the technical solution of the present invention, the system also includes a system monitoring and maintenance module, which is configured to monitor the health status of the system, application performance and blockchain status, use monitoring tools and alarm mechanisms to provide alarms and notifications, collect performance data, and perform log recording, tracking and analysis. Specifically, the system monitoring and maintenance module provides real-time visualization of system status through monitoring tools and dashboards, collects and analyzes system logs, resource usage data and response time indicators; detects anomalies by setting thresholds and rules, generates alarms and notifications, stores log data in a centralized manner, and provides query and analysis functions.

[0012] The system monitoring and maintenance module is a key component of the malicious crawler identification system based on blockchain. It is responsible for real-time monitoring of the system's operating status, performance indicators, and maintaining the stability and security of the system. This module ensures that the system can operate reliably under high load and complex environments, and can detect and solve problems in a timely manner.

[0013] As a further limitation of the technical solution of the present invention, the malicious crawler detection module includes a network traffic monitoring unit, a behavior analysis unit, an anomaly detection unit and a report generation unit; Network traffic monitoring unit, which captures all traffic data in the network; Behavior analysis unit, which analyzes the network behavior of users or devices by parsing network traffic data; Anomaly detection unit, which automatically detects abnormal behavior in the network; The report generation unit generates detailed detection reports and takes corresponding measures to deal with malicious crawlers.

[0014] The malicious crawler detection module is the core part of the malicious crawler identification system based on blockchain. Its main function is to monitor and analyze network traffic to identify and handle possible malicious crawler activities. The malicious crawler detection module is the core of the entire system, responsible for capturing, analyzing and identifying malicious crawler activities in the network. Through network traffic monitoring, behavior analysis and anomaly detection technology, the module can effectively identify potential crawler behaviors and generate detailed detection reports. Faced with challenges such as large data volume, behavioral diversity and camouflage technology, the system adopts efficient data processing technology and advanced detection algorithms to ensure the accuracy and real-time performance of detection.

[0015] As a further limitation of the technical solution of the present invention, the data recording module stores the detection results in a database or blockchain, and records the operation log, error log and system log, and stores them in the database or blockchain, provides a data query interface, and supports retrieval by time, IP address, and behavior feature conditions. Specifically, the data synchronization and sharing module regularly updates the data of each node through a data synchronization mechanism, provides a data sharing interface and protocol, allows each component to access and use shared data, and distributes the latest data to all relevant nodes through a data update notification mechanism.

[0016] As a further limitation of the technical solution of the present invention, the smart contract module writes detection rules in the smart contract, and the smart contract automatically distributes rewards according to preset rules and conditions, and stores the detection results and operation records on the blockchain.

[0017] The smart contract module is a key component of the malicious crawler identification system based on blockchain, responsible for automatically executing and managing malicious crawler detection rules and response measures. It uses the immutability and automation of blockchain to ensure the transparency and accuracy of detection and response.

[0018] As a further limitation of the technical solution of the present invention, the blockchain module maintains a consistent copy of the ledger on multiple nodes through blockchain technology, and ensures that once the data is written into the blockchain, it cannot be modified through encryption algorithms and chain data structures; deploys and runs smart contracts on the blockchain to automatically execute rules and response measures. The blockchain module is a core component of the malicious crawler identification system based on blockchain, responsible for providing distributed storage, immutability, transparency and security of data. It ensures that all data records and operations of the system can be reliably saved and verified.

[0019] It can be seen from the above technical solutions that the present invention has the following advantages: the blockchain system combines decentralization and smart contracts to provide more accurate detection rules and improve detection accuracy through the collaboration of multiple nodes. Smart contracts can automatically execute predefined detection rules and response measures, reduce manual intervention, and improve detection efficiency. Nodes in a decentralized network can process large-scale network traffic in parallel, thereby improving detection speed. Through a decentralized node network, the risk of single point failure is reduced, and the reliability and anti-attack capability of the system are improved. Even if some nodes are attacked or fail, other nodes can still operate normally to ensure the continued stability of the system. The records on the blockchain are public and cannot be tampered with, which means that all detection results and treatment measures are transparent. Network participants can view and verify these records to improve the credibility and transparency of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0021] Figure 1 is a schematic block diagram of a system according to an embodiment of the present invention. DETAILED DESCRIPTION

[0022] In order to enable those skilled in the art to better understand the technical solutions in the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0023] like Figure 1 As shown, an embodiment of the present invention provides a malicious crawler identification system based on blockchain, including a malicious crawler detection module, a data recording module, a smart contract module, a blockchain module, and a data synchronization and sharing module; The malicious crawler detection module is configured to capture network traffic and request information, define detection rules, implement a rule engine to mark potential malicious requests, and extract features for training machine learning models to identify normal traffic and crawler behavior; The data recording module is designed to collect the output results of the malicious crawler detection module and record these data in a standard format; The smart contract module includes smart contracts for data storage, query and record update. The contracts are compiled and deployed on the blockchain network and support interaction with the blockchain through contract methods. The blockchain module selects and configures the blockchain platform, designs and implements smart contracts, handles the combination of on-chain data and off-chain data storage, and monitors the operating status of the blockchain network; The data synchronization and sharing module is configured to implement real-time synchronization and periodic synchronization tasks when new data or data changes are detected, and provides data access services through the API interface.

[0024] The malicious crawler detection module captures network traffic and request information, defines detection rules, and combines machine learning models to identify normal traffic and crawler behavior, which can greatly improve the accuracy of malicious crawler identification. Compared with traditional identification methods, the system can more accurately identify malicious crawler behavior and reduce false positives and false negatives.

[0025] Using blockchain technology to record and store detection data can ensure the immutability and transparency of the data. This helps prevent malicious users from tampering with or deleting detection data, thereby protecting the legitimate rights and interests of website operators. The configuration of the data synchronization and sharing module enables the system to synchronize detection data in real time and regularly, and provide data access services through the API interface. This helps to achieve data sharing and collaborative work between different components, and improve the overall performance and efficiency of the system. The automation of data storage, query and record update through the smart contract module can reduce the labor cost and time cost of system operation. At the same time, the decentralized characteristics of blockchain technology also help to reduce the maintenance cost of centralized servers. The system adopts a modular design, and each module is relatively independent, and can be flexibly expanded and upgraded according to actual needs. This helps to adapt to future changes in the network environment and the development of malicious crawler behavior, and maintain the advancement and practicality of the system.

[0026] In some embodiments, the malicious crawler detection module includes a network traffic monitoring unit, a behavior analysis unit, an anomaly detection unit, and a report generation unit; Network traffic monitoring unit, which captures all traffic data in the network; Behavior analysis unit, which analyzes the network behavior of users or devices by parsing network traffic data; Anomaly detection unit, which automatically detects abnormal behavior in the network; The report generation unit generates detailed detection reports and takes corresponding measures to deal with malicious crawlers.

[0027] The malicious crawler detection module is the core part of the malicious crawler identification system based on blockchain. Its main function is to monitor and analyze network traffic to identify and handle possible malicious crawler activities. The malicious crawler detection module is the core of the entire system, responsible for capturing, analyzing and identifying malicious crawler activities in the network. Through network traffic monitoring, behavior analysis and anomaly detection technology, the module can effectively identify potential crawler behaviors and generate detailed detection reports. Faced with challenges such as large data volume, behavioral diversity and camouflage technology, the system adopts efficient data processing technology and advanced detection algorithms to ensure the accuracy and real-time performance of detection.

[0028] Configure web servers (e.g., Apache, Nginx) and application logging systems to ensure that detailed request information is captured. Use tools such as Wireshark, tcpdump, or commercial traffic monitoring solutions to capture network traffic. Define detection rules, such as request frequency, specific User-Agent, IP address blacklist, etc. Implement a rule engine to apply these rules and flag potentially malicious requests. Clean and process the collected data for model training. Extract useful features, such as request time interval, access path, etc. Train models using classification algorithms (e.g., K-means, decision trees, deep learning, etc.) to identify normal traffic and crawler behavior. Evaluate the accuracy and performance of the model using cross-validation and other techniques. Implement a real-time data processing pipeline to apply detection rules and machine learning models. Define response measures, such as blocking IP addresses, limiting request frequency, notifying administrators, etc. Integrate the detection module into an existing web application or network security system. Perform comprehensive testing, including simulating malicious crawler activities, performance testing, and security testing.

[0029] It should be noted that the steps of training anomaly detection models using historical data and known malicious crawler samples can be broken down into the following key steps: Collect historical data from sources such as network traffic logs and server access logs. Ensure that the data covers normal user behavior and known malicious crawler behavior. Remove invalid, duplicate, or redundant data records. Handle missing values, such as through interpolation, filling, or deletion. Extract features useful for anomaly detection from the raw data, such as request frequency, request type, request source, User-Agent, etc. Standardize or normalize the features to improve the training effect of the model. Divide the data into training set, validation set, and test set. The training set is used to train the model, the validation set is used to adjust the model parameters, and the test set is used to evaluate the model performance. According to the complexity of the problem and the characteristics of the data, select a suitable anomaly detection algorithm, such as statistical-based methods, machine learning classifiers, deep learning models, etc. Use the training set data to train the anomaly detection model.

[0030] During the training process, continuously adjust model parameters to optimize model performance. Use validation set data to validate the model and evaluate the accuracy and stability of the model. Adjust model parameters or select other algorithms for training based on the validation results. Select the most useful features for anomaly detection by analyzing the performance of the model on the validation set. Remove features that have little impact on model performance to reduce computational complexity and improve model generalization. Tune model hyperparameters such as learning rate, number of iterations, regularization parameters, etc. Use grid search, random search, or Bayesian optimization to adjust parameters. Fuse multiple anomaly detection models to improve model accuracy and robustness. Model fusion can be performed using voting, weighted average, or stacking. Use test set data to test the model and evaluate its accuracy and practicality. Analyze the performance of the model on the test set to determine whether it meets business requirements. Deploy the trained anomaly detection model to the production environment. Configure monitoring and alarm systems to notify relevant personnel in a timely manner when malicious crawler behavior is detected. Continuously monitor the deployed model and evaluate its performance in actual applications. Update and optimize the model regularly according to business needs and data changes.

[0031] In some embodiments, the data recording module stores the detection results in a database or blockchain, and records the operation log, error log and system log, and stores them in the database or blockchain, provides a data query interface, and supports retrieval by time, IP address, and behavior feature conditions. Specifically, the data synchronization and sharing module regularly updates the data of each node through a data synchronization mechanism, provides a data sharing interface and protocol, allows each component to access and use shared data, and distributes the latest data to all relevant nodes through a data update notification mechanism.

[0032] The data recording module is a crucial component of the malicious crawler identification system based on blockchain, responsible for collecting, storing and managing detected data. This module ensures that all detection results and related information are accurately recorded and saved, providing a reliable data basis for subsequent analysis and auditing. The implementation method is: Store the detection results (such as the identification of malicious crawler activities, detection time, etc.) in the database or blockchain. Save the crawler's behavioral data, including access patterns, request frequency, source IP and other information. Record operation logs, error logs and system logs, and store them in the database or blockchain. Provide a data query interface to support retrieval by time, IP address, behavioral characteristics and other conditions.

[0033] Collect the output of the malicious crawler detection module, such as IP address, detection time, behavioral characteristics, risk score, etc. Design a standard data format (such as JSON or XML) to store detection results and additional information. For example: { "ip_address": "192.168.1.1", "detection_time": "2024-08-25T14:30:00Z", "behavior_features": { "request_frequency": 100, "page_access_pattern":"abnormal", "user_agent": "crawler_bot"}, "risk_score": 85}.

[0034] Since the blockchain has limited storage space, data can be compressed or detailed information can be recorded using hash values. Choose a blockchain platform: Ethereum: supports smart contracts and decentralized applications (dApps), suitable for complex data storage and operations. Hyperledger Fabric: suitable for enterprise-level applications, providing permission management and privacy protection functions. Polygon: suitable for high-throughput scenarios, compatible with the Ethereum ecosystem. Configure blockchain nodes to ensure network reliability and data synchronization. Write a smart contract to store malicious crawler detection records. The contract should support inserting, querying, and updating records. Define the data structure and storage format to ensure the integrity of the record. Submit the formatted data to the smart contract for recording. Ensure the integrity and immutability of the data. Process blockchain transactions, record data, and wait for block confirmation.

[0035] Consider transaction fees and processing time. If the data is large, the data hash value can be stored on the blockchain, and the actual data can be stored in off-chain storage (such as IPFS). Implement query functions to query records stored on the blockchain through smart contracts. Retrieval can be based on fields such as IP address and detection time. Provide API interfaces or user interfaces for administrators and system integrators to view and retrieve records. Through these technical implementation details, an efficient data recording module can be built to permanently record malicious crawler detection results on the blockchain to ensure data security and transparency.

[0036] In some embodiments, the smart contract module writes detection rules in a smart contract, and the smart contract automatically distributes rewards according to preset rules and conditions, and stores the detection results and operation records on the blockchain.

[0037] The smart contract module is a key component of the malicious crawler identification system based on blockchain, responsible for automatically executing and managing malicious crawler detection rules and response measures. It uses the immutability and automation of blockchain to ensure the transparency and accuracy of detection and response.

[0038] Write detection rules in smart contracts, and smart contracts automatically execute these rules when specific conditions are met. Define specific business logic and operation processes in smart contracts, such as recording detection results, updating data status, etc. When specific conditions are met, smart contracts automatically trigger response measures without human intervention. Smart contracts automatically distribute rewards based on preset rules and conditions. Store important detection results and operation records on the blockchain to ensure that they are tamper-proof and traceable.

[0039] Record the detection results of malicious crawlers, including IP address, detection time, behavioral characteristics, and risk score. Allow querying of stored records through contract methods. Support updating existing records (if necessary). Compile the contract using the Solidity compiler to generate bytecode and ABI. Deploy the contract to a blockchain network (such as the Ethereum mainnet or testnet). An online IDE for writing, compiling, and deploying Solidity contracts. Truffle / Hardhat: A development framework for local development, testing, and deployment of contracts. Interact with smart contracts using the Web3 library or the Ethereum SDK. When calling a contract function (such as addRecord), build and send a transaction and wait for transaction confirmation. Operations on the blockchain usually require transaction fees, so ensure that the budget is sufficient. Consider encrypting sensitive data to protect user privacy. Consider the scalability of the system when designing the contract to support future feature additions. With these technical implementation details, an efficient and secure smart contract module can be built to store and manage malicious crawler detection records, using blockchain technology to ensure data transparency and immutability.

[0040] In some embodiments, the blockchain module maintains a consistent copy of the ledger on multiple nodes through blockchain technology, and ensures that once the data is written to the blockchain, it cannot be modified through encryption algorithms and chain data structures; deploys and runs smart contracts on the blockchain to automatically execute rules and response measures. The blockchain module is a core component of the malicious crawler identification system based on blockchain, responsible for providing distributed storage, immutability, transparency and security of data. It ensures that all data records and operations of the system can be reliably saved and verified.

[0041] Blockchain technology is used to maintain a consistent copy of the ledger on multiple nodes to ensure distributed storage and consistency of data. Encryption algorithms and chain data structures are used to ensure that once data is written to the blockchain, it cannot be modified. All data and operations on the blockchain can be accessed and verified by all nodes and recorded on the chain. Smart contracts are deployed and run on the blockchain to automatically execute rules and response measures. Consensus algorithms (such as PoW, PoS, BFT) are used to reach consensus and ensure data reliability. Encryption algorithms (such as hash functions, public key encryption) are used to ensure data security and privacy.

[0042] Select the appropriate blockchain platform according to the needs, and perform network configuration and node deployment. Design and implement smart contracts, compile and deploy them to the blockchain network. Record malicious crawler detection data into smart contracts. Process the combination of on-chain data and off-chain data storage. Query data through smart contracts, support retrieval and management of stored records. Monitor the operating status of the blockchain network and perform necessary maintenance and upgrades. Through these technical implementation details, a stable and reliable blockchain module can be effectively built to ensure the secure storage and management of malicious crawler detection data.

[0043] In some embodiments, the system further includes: an incentive mechanism module, which designs and deploys smart contracts, sets incentive rules, and encourages users to participate in malicious crawler detection. Specifically, the incentive mechanism module automatically distributes rewards according to the contributions of participants, automatically calculates the reward amount through smart contracts, sets incentive rules in smart contracts, stores incentive records on the blockchain, collects feedback data from participants, and adjusts incentive rules based on feedback information.

[0044] The incentive mechanism module is an important part of the malicious crawler identification system based on blockchain. It automatically distributes rewards based on the contributions of participants (such as submitting detection results and providing data). The reward amount is automatically calculated through smart contracts to ensure the transparency and accuracy of reward calculation. The incentive rules are set in the smart contract and adjusted and updated according to actual conditions. The incentive records (such as reward distribution, participant contributions, etc.) are stored on the blockchain. The feedback data of participants is collected and the incentive rules are adjusted based on the feedback information.

[0045] Design and deploy smart contracts. Distribute and set reward rules to early participants. Develop a points calculation and rating mechanism. When users submit data or participate in detection, the smart contract issues rewards and points according to the rules. Monitor system activities, and regularly evaluate and adjust incentive strategies. Regularly hold competitions and challenges to encourage users to actively participate. Provide a user feedback mechanism to collect community opinions on the incentive mechanism. Through these technical implementation details, an incentive mechanism module can be effectively built to promote participation and data quality in the malicious crawler detection system, and improve the overall system operation efficiency and reliability.

[0046] When the data synchronization and sharing module detects new data or data changes, it triggers events or scheduled tasks for synchronization. Real-time synchronization: Use WebSocket to push data updates to all nodes. Perform data synchronization tasks regularly to distribute updated data to each node. Provide data access services to users and systems through API interfaces. Ensure that data remains encrypted during transmission and storage to protect data privacy. Use consistency protocols to resolve data conflicts in distributed environments. Regularly check and fix data inconsistencies. Through these technical implementation details, an efficient data synchronization and sharing module can be built to ensure real-time update and reliable sharing of data in the malicious crawler detection system, and maintain data consistency and security.

[0047] The system also includes a system monitoring and maintenance module, which is configured to monitor the health status of the system, application performance, and blockchain status, use monitoring tools and alarm mechanisms to provide alarms and notifications, collect performance data, and perform log recording, tracking, and analysis. Specifically, the system monitoring and maintenance module provides real-time visualization of system status through monitoring tools and dashboards, collects and analyzes system logs, resource usage data, and response time indicators; detects anomalies by setting thresholds and rules, generates alarms and notifications, stores log data in a centralized manner, and provides query and analysis functions.

[0048] Provide real-time visualization of system status through monitoring tools and dashboards. Collect and analyze system logs, resource usage data, response time and other indicators. Detect anomalies by setting thresholds and rules, and generate alerts and notifications. Store log data in a centralized manner and provide query and analysis functions. Include software updates, database maintenance, backup and recovery, etc. Monitor security events, implement security policies, perform vulnerability scanning and repair. Set up a user support system to provide technical support and problem feedback mechanism.

[0049] Although the present invention has been described in detail with reference to the accompanying drawings and in combination with the preferred embodiments, the present invention is not limited thereto. Without departing from the spirit and essence of the present invention, a person of ordinary skill in the art may make various equivalent modifications or substitutions to the embodiments of the present invention, and these modifications or substitutions shall be within the scope of the present invention. Any person of ordinary skill in the art may easily think of changes or substitutions within the technical scope disclosed by the present invention, and these shall be within the scope of protection of the present invention.

Claims

1. A malicious crawler identification system based on blockchain, characterized in that: Including malicious crawler detection module, data recording module, smart contract module, blockchain module, data synchronization and sharing module; The malicious crawler detection module is configured to capture network traffic and request information, define detection rules, implement a rule engine to mark potential malicious requests, and extract features for training machine learning models to identify normal traffic and crawler behavior; The data recording module is designed to collect the output results of the malicious crawler detection module and record these data in a standard format; The smart contract module includes smart contracts for data storage, query and record update. The contracts are compiled and deployed on the blockchain network and support interaction with the blockchain through contract methods. The blockchain module selects and configures the blockchain platform, designs and implements smart contracts, handles the combination of on-chain data and off-chain data storage, and monitors the operating status of the blockchain network; The data synchronization and sharing module is configured to implement real-time synchronization and periodic synchronization tasks when new data or data changes are detected, and provides data access services through the API interface.

2. The malicious crawler identification system based on blockchain according to claim 1 is characterized in that: The system also includes: an incentive mechanism module, which designs and deploys smart contracts, sets incentive rules, and encourages users to participate in malicious crawler detection.

3. The malicious crawler identification system based on blockchain according to claim 2 is characterized in that: The system also includes a system monitoring and maintenance module, which is configured to monitor the health status of the monitoring system, application performance and blockchain status, use monitoring tools and alarm mechanisms to provide alarms and notifications, collect performance data, and perform log recording, tracking and analysis.

4. The malicious crawler identification system based on blockchain according to claim 3 is characterized in that: The malicious crawler detection module includes a network traffic monitoring unit, a behavior analysis unit, an anomaly detection unit, and a report generation unit; Network traffic monitoring unit, capturing all traffic data in the network Behavior analysis unit, which analyzes the network behavior of users or devices by parsing network traffic data; Anomaly detection unit, which automatically detects abnormal behavior in the network; The report generation unit generates detailed detection reports and takes corresponding measures to deal with malicious crawlers.

5. The malicious crawler identification system based on blockchain according to claim 4 is characterized in that: The data recording module stores the detection results in the database or blockchain, and records the operation log, error log and system log, and stores them in the database or blockchain, providing a data query interface and supporting retrieval by time, IP address and behavioral feature conditions.

6. The malicious crawler identification system based on blockchain according to claim 5 is characterized in that: The smart contract module writes detection rules in the smart contract. The smart contract automatically distributes rewards based on preset rules and conditions, and stores the detection results and operation records on the blockchain.

7. The malicious crawler identification system based on blockchain according to claim 6 is characterized in that: The blockchain module uses blockchain technology to maintain a consistent copy of the ledger on multiple nodes. Through encryption algorithms and chain data structures, it ensures that once the data is written into the blockchain, it cannot be modified. Deploy and run smart contracts on the blockchain to automate rules and response measures.

8. The malicious crawler identification system based on blockchain according to claim 7 is characterized in that: The incentive mechanism module automatically distributes rewards based on the contributions of participants, automatically calculates the reward amount through smart contracts, sets incentive rules in smart contracts, stores incentive records on the blockchain, collects feedback data from participants, and adjusts incentive rules based on feedback information.

9. The malicious crawler identification system based on blockchain according to claim 8, characterized in that: The data synchronization and sharing module regularly updates the data of each node through the data synchronization mechanism, provides data sharing interfaces and protocols, allows each component to access and use shared data, and distributes the latest data to all relevant nodes through the data update notification mechanism.

10. The malicious crawler identification system based on blockchain according to claim 9 is characterized in that: The system monitoring and maintenance module provides real-time visualization of system status through monitoring tools and dashboards, collects and analyzes system logs, resource usage data, and response time indicators; detects anomalies by setting thresholds and rules, generates alerts and notifications, stores log data in a centralized manner, and provides query and analysis capabilities.

Images