An anti-crawler method, device, equipment and medium

Abstract

The application relates to an anti-crawler method, device, equipment and medium. In the application, historical access data of a website is acquired, the historical access data is classified and marked based on a pre-trained crawler data recognition model to obtain crawler data and non-crawler data; in response to a first access request initiated by a client, the crawler data is encrypted to obtain ciphertext crawler data, and the non-crawler data and the ciphertext crawler data are returned to the client; in response to a second access request initiated by the client based on the ciphertext crawler data, an access code corresponding to the ciphertext crawler data and a request user is generated and delivered to a user terminal, and an authentication triple including a ciphertext identifier, a user identifier and the access code is constructed; and in response to a third access request initiated by the client, verification information carried by the third access request is detected according to the authentication triple, and when the detection result is passed, a decryption key and a decryption algorithm are delivered to the client.

Description

Technical Field

[0001] This invention relates to the field of anti-crawler technology, specifically to an anti-crawler method, apparatus, equipment, and medium. Background Technology

[0002] With the widespread adoption of internet information services and the rapid development of the artificial intelligence big data model industry, the massive amounts of publicly available data carried by the Web (World Wide Web) platform possess extremely high commercial value and data asset attributes. The illegal crawling of website data by batches is becoming increasingly frequent, not only causing data asset loss for website operators but also congesting the access bandwidth of normal users and interfering with the stable operation of platform services. Anti-crawler technology is a type of protective technology used by website operators to identify, block, or restrict illegal crawler access, ensuring their own data security and the normal operation of services. It is an important research direction in the current field of Web data security.

[0003] Current anti-scraping solutions mostly build their protection systems around the backend server. A typical implementation uses an access token-based verification mechanism: after receiving an access request from a client, the server first checks if the request carries a valid access token. For requests without a token, it first identifies whether the request is malicious. If not malicious, an access token is generated and normal data is returned; if malicious, error data is returned. For requests with a token, the token is cross-validated. If the verification passes, normal data is returned; if the verification fails, error data is returned. In addition, existing technologies have also developed various backend anti-scraping methods, such as IP (Internet Protocol address) access frequency restrictions, request header feature matching, and user behavior trajectory analysis, to identify and intercept crawler requests from different dimensions.

[0004] However, the existing anti-crawler methods generally suffer from the technical problem of having a single protection dimension: the anti-crawler verification logic is all concentrated on the backend, and the frontend does not have a corresponding deep protection mechanism. The generation and transmission logic of verification credentials such as access tokens can be cracked by reverse engineering the frontend JavaScript (Web Frontend Scripting Language) code. The crawler program can simulate and generate legitimate verification credentials based on the cracked logic, thereby bypassing the backend verification mechanism and directly obtaining the complete target data. This causes the anti-crawler protection to fail and cannot effectively resist professional crawler attacks with reverse engineering capabilities, posing a serious data security risk.

[0005] Therefore, how to improve the ability to resist reverse engineering during anti-scraping operations to compensate for the security deficiencies of single-backend protection is an urgent problem to be solved. Summary of the Invention

[0006] The technical problem to be solved by this invention is how to improve the anti-reverse engineering capability during anti-crawling to make up for the security defects of single backend protection.

[0007] The technical solution of the present invention to solve the above-mentioned technical problems is as follows: This application provides an anti-crawler method, including: Historical access data of a website is obtained, and the historical access data is classified and labeled based on a pre-trained crawler data recognition model to obtain crawler data and non-crawler data. The crawler data is website data accessed by crawler tools, and the non-crawler data is website data accessed by normal web page clients. In response to the first access request initiated by the client, the crawler data is encrypted to obtain ciphertext crawler data, and the non-crawler data and the ciphertext crawler data are returned to the client. In response to the second access request initiated by the client based on the encrypted crawler data, an access code corresponding to the encrypted crawler data and the requesting user is generated and sent to the user terminal, and an authentication triplet including the encrypted identifier, the user identifier and the access code is constructed. In response to a third access request initiated by the client, the authentication information carried in the third access request is checked according to the authentication triplet. If the check result is successful, a decryption key and decryption algorithm are sent to the client so that the client can decrypt the encrypted crawler data according to the decryption key and the decryption algorithm to obtain the plaintext crawler data.

[0008] The beneficial effects of this invention are as follows: Based on a pre-trained crawler data recognition model, historical website access data is classified and labeled to distinguish between crawler data and non-crawler data. Encryption is applied only to core data frequently accessed by crawlers. This achieves precise protection while converting the core target data of crawlers into ciphertext. Furthermore, in response to the first access request, only plaintext non-crawler data and ciphertext crawler data are returned to the client. No decryption keys, decryption algorithms, or other core decryption credentials are issued throughout the process. This ensures that even if a crawler reverse-engineers the JavaScript code of the front-end page, it cannot obtain valid information for directly decrypting data from the front-end logic and returned data during the initial interaction. This avoids the problem in traditional anti-crawler schemes where the token generation logic is exposed on the front-end and can be reverse-engineered. The core authentication credentials are completely isolated from the webpage front-end link. In response to the second access request, the server generates a dynamic access code that is uniquely bound to the corresponding ciphertext crawler data and the requesting user. This access code is then sent to an independent user terminal. Simultaneously, this process is only performed on the server side. The system locally constructs an authentication triplet containing a ciphertext identifier, a user identifier, and an access code. The generation logic and storage of the access code are controlled by the server, and the distribution channel is independent of the web front-end code execution environment. Crawlers cannot intercept, deduce, or simulate the generation of this access code by reverse engineering the front-end code. This solves the defect of single-backend protection, where the verification credential relies entirely on front-end interaction and is easily reverse engineered. The distribution of the core decryption credential is postponed until after multi-step verification is completed. The corresponding decryption key and decryption algorithm are only distributed to the client after responding to the third access request and the authentication triplet is detected. Obtaining decryption capability requires completing multi-step authentication across terminals, and the decryption credential is strongly bound to the specific ciphertext and the specific user. Crawlers cannot directly obtain data decryption permissions by bypassing front-end verification or forging a single request. Compared with single-backend protection solutions that only rely on back-end token verification, this significantly increases the technical threshold and implementation cost for crawler reverse engineering, effectively making up for the security defects of single-backend protection, where the front-end logic is easily reverse engineered and the verification credential is easily replicated.

[0009] Based on the above technical solution, the present invention can be further improved as follows.

[0010] Furthermore, the process of obtaining historical website access data involves classifying and labeling the historical access data based on a pre-trained crawler data recognition model to obtain crawler data and non-crawler data, including: The percentage of times each training data point was accessed by the crawler tool was calculated out of the total number of accesses, thus obtaining the crawler access percentage value corresponding to each training data point. The crawler access ratio of each access training data is compared with a preset ratio threshold. The access training data with a ratio greater than the preset ratio threshold is labeled with crawler data labels, and the access training data with a ratio less than or equal to the preset ratio threshold is labeled with non-crawler data labels, thus obtaining an access training dataset with binary labels. Based on the access training dataset with binary labels, the pre-built crawler data recognition model is trained under supervision to obtain the trained crawler data recognition model. The trained web crawler data recognition model is used to predict and classify the historical access data and label it accordingly, thus obtaining the web crawler data and the non-web crawler data.

[0011] The beneficial effects of adopting the above-mentioned further solution are as follows: First, the percentage of crawler accesses is statistically analyzed. Then, training samples are constructed by comparing and labeling binary data with thresholds. Using objective access data as the labeling basis avoids subjective errors in manual labeling and ensures the accuracy and consistency of training sample labeling. Then, the crawler data recognition model is trained through supervised training, enabling the model to accurately learn the characteristic patterns of crawler data. Finally, the trained model automatically predicts and classifies historical access data, which can stably and efficiently complete the classification and labeling of large amounts of website data. This not only improves the accuracy of distinguishing between crawler data and non-crawler data, making subsequent encryption protection more targeted and avoiding non-core data being additionally encrypted and affecting the browsing experience of normal users, but also reduces the operational costs of manual classification and provides a reliable data classification foundation for subsequent differentiated anti-crawler protection.

[0012] Furthermore, the encryption process for the crawler data to obtain ciphertext crawler data includes: The crawler data is parsed to obtain structured data with key-value pairs where key names and value contents correspond one-to-one. The key names in the structured data are kept in plaintext without processing. Symmetric encryption is performed on the numerical content corresponding to each key name in the structured data to obtain the encrypted numerical content. The plaintext key name and the encrypted numerical content are combined according to a preset data structure to obtain the encrypted crawler data.

[0013] The beneficial effects of adopting the above-mentioned further solution are as follows: First, the key-value pair structured data is parsed, and then a selective encryption method is adopted, which preserves the plaintext of the key names and encrypts the numerical content. On the one hand, it accurately protects the core business content that the crawler is interested in, so that even if the crawler successfully obtains the interface response data, it cannot extract valid business information, thus ensuring data security at the content level. On the other hand, it preserves the structural identifier of the data, allowing the front-end to normally parse the data structure and render the basic page layout. The encryption operation will not affect the basic page browsing experience of normal users. At the same time, encrypting only the core numerical content reduces the amount of data processing for encryption operations. Combined with the efficient processing characteristics of symmetric encryption operations, it effectively reduces the server performance overhead of the encryption process and can adapt to the high-concurrency access scenarios of the website. In addition, the encrypted data is reconstructed according to the preset data structure, ensuring high compatibility of the data format before and after encryption. There is no need to carry out large-scale transformation of the front-end data parsing logic, which greatly reduces the implementation cost of the solution.

[0014] Furthermore, the construction of the authentication triplet, which includes a ciphertext identifier, a user identifier, and an access code, includes: The encrypted values ​​corresponding to each key in a single encrypted crawler data are concatenated in a preset order to obtain a continuous string. The continuous string is then subjected to one-way message digest encoding to obtain a ciphertext identifier that corresponds one-to-one with the encrypted crawler data. Extract the user's mobile phone number carried in the second access request, and use the user's mobile phone number as the user identifier of the authentication triplet; The ciphertext identifier, the user identifier, and the access code are combined to obtain the authentication triplet.

[0015] The beneficial effects of adopting the above-mentioned further scheme are as follows: By generating ciphertext identifiers through concatenating ciphertext values ​​and then performing one-way message digest encoding, it ensures that each piece of ciphertext crawler data has a unique identity identifier, accurately corresponding to specific ciphertext data and decryption credentials. Furthermore, the irreversible nature of one-way encoding prevents the ciphertext identifier from leaking the original content information of the ciphertext. Simultaneously, the fixed identifier length facilitates rapid indexing and querying by the server. Using the user's mobile phone number as the user identifier achieves unique binding of the user's identity and unifies the receiving terminal of the identity identifier and access code, ensuring that the access code is accurately delivered to the real user and preventing crawlers from impersonating anonymous identities to request decryption permissions. Finally, the ciphertext identifier, user identifier, and dynamic access code are combined into an authentication triplet, with each three bound one-to-one. This makes the verification dimensions for each decryption request more comprehensive, verifying both the user's identity and the matching relationship between the access code and the corresponding ciphertext. This avoids the problem of single-dimensional verification being easily bypassed, effectively improving the accuracy and security of identity verification and providing a reliable verification foundation for the subsequent issuance of decryption credentials.

[0016] Furthermore, the method also includes: In response to the first access request initiated by the client, the non-crawler data and the encrypted crawler data are returned to the client, without issuing any decryption credentials to the client; In response to the second access request initiated by the client based on the encrypted crawler data, an access code is generated and sent to the user terminal, but the decryption key and decryption algorithm are not sent. In response to the third access request initiated by the client, after completing the verification information detection, the decryption key and decryption algorithm are sent to the client.

[0017] The beneficial effects of adopting the above-mentioned further solution are as follows: By using a phased and tiered credential distribution mechanism, the core decryption credential is completely isolated from the basic data requests and verification application requests in terms of timing. The first request only distributes the data itself, the second request only distributes the intermediate verification credential, and the decryption key and decryption algorithm are only distributed in the third stage after the completion of full identity verification. This completely avoids the problem of the decryption credential and data being returned synchronously in the traditional solution, which is easily intercepted by a single interface request. This makes it impossible for crawlers to obtain the complete decryption capability by reverse engineering the front-end code or simulating a single interface request, thus greatly improving the security of the core decryption credential. At the same time, the layered distribution mechanism ensures that each interaction only exposes the information of the corresponding level, effectively reducing the exposure scope of sensitive information. From the perspective of credential control, it strengthens the overall anti-crawler system's anti-reverse engineering capability and further makes up for the security defects of easy credential leakage and single protection level in single back-end protection.

[0018] Furthermore, the step of detecting the verification information carried in the third access request based on the authentication triplet includes: Extract the user identifier and access code from the authentication triplet as the first verification information; Extract the user identifier and access code from the third access request as the second verification information; The first verification information is compared with the second verification information to obtain the information comparison result; If the information comparison results show that the information matches, the detection result is determined to be passed.

[0019] The beneficial effects of adopting the above-mentioned further solution are as follows: The verification logic employs a two-dimensional comparison between server-side baseline information and client-side verification information, using locally stored authentication triples as the verification benchmark. The verification standard is never exposed externally throughout the process, fundamentally preventing the risk of verification rules being reverse-engineered by the front-end or the verification benchmark being tampered with. Simultaneously, dual verification is performed on both the user identifier and the access code, verifying both the legitimacy of the user's identity and the binding relationship between the access code and the user. This prevents vulnerabilities where a single piece of information can be misused to bypass verification, effectively improving the anti-forgery capability and accuracy of identity verification. Furthermore, the item-by-item comparison logic has high computational efficiency, enabling rapid verification in high-concurrency scenarios. While ensuring strong security protection, it also considers the user's access experience, providing a reliable basis for the secure issuance of decryption credentials and further strengthening the overall anti-crawler system's reliability.

[0020] Furthermore, the encryption process for the crawler data to obtain ciphertext crawler data includes: For business scenarios where the data security level is higher than a preset level threshold and the concurrency is lower than a preset concurrency threshold, the crawler data is encrypted in real time after receiving the first access request to generate the encrypted crawler data. For business scenarios where the concurrency exceeds a preset concurrency threshold or the access volume exceeds a preset access volume threshold, the crawler data is encrypted before the first access request is received, generating the ciphertext crawler data and storing the ciphertext crawler data in the ciphertext database.

[0021] The beneficial effects of adopting the above-mentioned further solutions are as follows: Business scenarios are divided based on three dimensions: data security level, concurrency, and access volume. Two different encryption processing times—real-time encryption and pre-encrypted storage—are specifically matched. For high-security, low-concurrency scenarios, request-triggered real-time encryption is used, allowing for the configuration of independent encryption parameters for each request. This enhances the dynamism of encryption and the difficulty of cracking, maximizing the security of high-value data. For high-concurrency, high-access scenarios, a pre-encrypted storage solution is used, shifting the encryption computation load from the request-response phase to idle business periods. This avoids encryption computation slowing down system response speed under high concurrency, effectively improving the system's concurrency capacity and access stability. The two encryption strategies can be flexibly selected and switched according to the actual needs of different businesses. This balances different levels of data security protection requirements with the trade-off between security protection and system performance, avoiding the limitations of a single encryption mode being unable to adapt to diverse business scenarios. This effectively improves the scenario adaptability and practicality of the entire anti-crawler solution.

[0022] This application embodiment also provides an anti-crawler device, the device comprising: The acquisition module is used to acquire historical access data of the website. Based on a pre-trained crawler data recognition model, the historical access data is classified and labeled to obtain crawler data and non-crawler data. The crawler data is website data accessed by crawler tools, and the non-crawler data is website data accessed by normal web page clients. An encryption module is used to encrypt the crawler data in response to a first access request initiated by the client, obtain ciphertext crawler data, and return the non-crawler data and the ciphertext crawler data to the client. The construction module is used to respond to the second access request initiated by the client based on the encrypted crawler data, generate an access code corresponding to the encrypted crawler data and the requesting user and send it to the user terminal, and construct an authentication triple including the encrypted identifier, the user identifier and the access code; The detection module is used to respond to a third access request initiated by the client, and to detect the verification information carried in the third access request based on the authentication triplet. If the detection result is successful, the module sends a decryption key and a decryption algorithm to the client so that the client can decrypt the encrypted crawler data according to the decryption key and the decryption algorithm to obtain the plaintext crawler data.

[0023] This application also provides an electronic device, including a processor and a memory, wherein the memory stores a plurality of instructions; the processor loads instructions from the memory to execute steps in any of the anti-crawler methods provided in this application.

[0024] This application also provides a computer-readable storage medium storing a plurality of instructions adapted for loading by a processor to execute steps in any of the anti-crawler methods provided in this application. Attached Figure Description

[0025] Figure 1 This is a flowchart illustrating the anti-crawler method provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the anti-crawler device provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0026] The principles and features of the present invention are described below. The examples given are only for explaining the present invention and are not intended to limit the scope of the present invention.

[0027] This application provides an anti-crawler method, apparatus, electronic device, and medium.

[0028] Specifically, the anti-crawler device can be integrated into an electronic device, such as a terminal or server. The terminal can be a mobile phone, tablet, smart Bluetooth device, laptop, or personal computer (PC); the server can be a single server or a server cluster consisting of multiple servers.

[0029] In some embodiments, the anti-crawler device can also be integrated into multiple electronic devices, such as multiple servers, with multiple servers implementing the anti-crawler method of this application.

[0030] In some embodiments, the terminal can also be used as a server to implement some or all of the functions of a server.

[0031] In the embodiments of this application, the terms "module" or "unit" may refer to a computer program or part of a computer program with a predetermined function, which works together with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) may be used to implement one or more modules or units. Furthermore, each module or unit may be part of an overall module or unit that includes the functions of that module or unit.

[0032] The following sections provide detailed descriptions of each example. It should be noted that the order of the following embodiments is not intended to limit the preferred order of the embodiments.

[0033] This embodiment provides an anti-crawler method. See also... Figure 1 , Figure 1 This is a flowchart illustrating the anti-crawler method provided in an embodiment of the present invention, as shown below. Figure 1 As shown, the specific process of this anti-scraping method can be as follows: 110. Obtain historical access data of the website, and classify and label the historical access data based on a pre-trained crawler data recognition model to obtain crawler data and non-crawler data, wherein the crawler data is website data accessed by crawler tools, and the non-crawler data is website data accessed by normal web page clients. 120. In response to the first access request initiated by the client, the crawler data is encrypted to obtain ciphertext crawler data, and the non-crawler data and the ciphertext crawler data are returned to the client. 130. In response to the second access request initiated by the client based on the encrypted crawler data, generate an access code corresponding to the encrypted crawler data and the requesting user and send it to the user terminal, and construct an authentication triple including the encrypted identifier, the user identifier and the access code; 140. In response to the third access request initiated by the client, the authentication triplet is used to detect the verification information carried in the third access request. If the detection result is successful, a decryption key and a decryption algorithm are sent to the client so that the client can decrypt the encrypted crawler data according to the decryption key and the decryption algorithm to obtain the plaintext crawler data.

[0034] The crawler data identification model is a classification and prediction model obtained through supervised training using machine learning. The training process is based on historical website access data. First, the proportion of each data item accessed by the crawler tool is counted out of the total number of accesses. Data with a proportion exceeding a preset threshold is labeled as crawler data, and data with a proportion below the threshold is labeled as non-crawler data, forming a training dataset with binary labels. Then, the dataset is trained using machine learning algorithms to finally obtain a model that can automatically classify input website data and determine whether it belongs to crawler data or non-crawler data. This model is used to pre-classify and label all website data to achieve differentiated data protection.

[0035] Among them, crawler data refers to website data whose access sources are mainly crawler tools, as statistically analyzed by the website backend. In other words, it is website content data that crawler tools crawl frequently and that crawlers pay close attention to. This type of data is usually the core crawling target of large model training and data aggregation crawlers, and it is also the key target of anti-crawler protection. It will be marked with corresponding tags and encrypted to prevent crawlers from directly obtaining plaintext content.

[0036] Non-crawler data refers to website data whose access sources are mainly normal web page clients, as statistically analyzed by the website's backend. This is website content data accessed by ordinary users during normal browsing and rarely crawled by crawler tools. This type of data is not the core crawling target of crawlers, so there is no need for encryption protection. It is returned and displayed directly in plaintext form when users access it, ensuring the browsing experience of normal users.

[0037] The first access request is a request for page data retrieval initiated by the client to the server when the user browses the website page through the web client. It corresponds to the data interface request required for the rendering of the website page. The client obtains all the target data required for the page display from the server through this request. It is the initial interaction request of the entire anti-scraping process, which triggers the server to classify, process and encrypt the target data and return it.

[0038] Among them, encrypted crawler data is encrypted data obtained by encrypting plaintext crawler data. For website data with key-value pair structure, only the numerical content of the data is encrypted, while the key name part of the data is kept in plaintext form. Each piece of encrypted crawler data has a unique encrypted identifier. This data will be displayed on the client page in a garbled or unreadable form and cannot be directly parsed by crawler tools to obtain valid content.

[0039] The second access request is an access request initiated by the client to the server when the user needs to view the plaintext content of the data after the client receives the encrypted crawler data. The request carries the encrypted identifier of the corresponding encrypted crawler data and the user's identity identifier, which is used to trigger the server to generate and issue an access code. It is an intermediate interaction request connecting data display and identity verification.

[0040] The access code is a one-time dynamic verification character generated by the server after receiving the second access request, based on the corresponding encrypted data and the corresponding user. Each access code is bound to a single piece of encrypted data and a single user, and the access code generated for each request is different. The server will send it to the user's terminal device through an independent communication channel as the core credential for user identity verification. It cannot be generated by reverse engineering through the front-end web page code.

[0041] The user terminal is a user-held device that is independent of the client that initiates the webpage access. It is used to receive the access code issued by the server and is usually the user's mobile phone device. The access code is sent to the terminal through the SMS channel, so that the transmission link of the verification credential is separated from the running environment of the webpage front-end, thus preventing crawlers from intercepting the verification credential through front-end reverse engineering.

[0042] The authentication triplet is an identity verification data set built locally on the server side when the access code is generated. It consists of three elements: ciphertext identifier, user identifier, and access code. The three elements are bound one-to-one and serve as the verification basis for subsequent decryption requests. This data set is stored only on the server and is not sent to the front-end client to prevent the verification rules from being reverse-engineered.

[0043] The third access request is a decryption permission request initiated by the client to the server after the user obtains the access code and enters it in the web client. The request carries the user identifier and the access code entered by the user, which is used to trigger the server to verify the identity. It is the final interactive request to obtain the decryption certificate.

[0044] The verification information is the identity credential information carried in the third access request, which is used to compare and verify with the authentication triplet. Specifically, it includes two parts: user identifier and access code. The server compares the verification information carried in the request with the authentication triplet stored locally to determine whether the current request was initiated by a legitimate user, and thus decides whether to issue a decryption credential.

[0045] The decryption key is a decryption parameter that corresponds one-to-one with the key used when encrypting the crawler data. It is completely consistent with the key used in the encryption process and belongs to the core decryption parameter under the symmetric encryption system. It is bound to the ciphertext identifier of the corresponding ciphertext data and stored locally on the server. It is only sent to the client after the identity verification is passed, and is used to perform decryption operations on the ciphertext crawler data.

[0046] The decryption algorithm is the inverse operation algorithm corresponding to the encryption algorithm used when encrypting the crawler data. For the symmetric encryption algorithm used during encryption, the decryption algorithm is the corresponding symmetric decryption operation logic. Each piece of ciphertext data corresponds to a unique decryption algorithm, which is stored locally on the server along with the decryption key. After verification, it is sent to the client and used with the decryption key to complete the decryption operation of the ciphertext data.

[0047] Plaintext crawler data is the original website data obtained by decrypting ciphertext crawler data. The content is completely consistent with the crawler data before encryption and can be directly displayed on the web client for legitimate users to browse and view. Only users who have passed full identity verification can obtain the plaintext content.

[0048] For example, the specific methods for obtaining crawled data and non-crawled data can be as follows: First, historical access data accumulated on the website is collected. Then, a pre-trained crawler data recognition model is called. Each piece of historical access data is input into the model for prediction and classification. The model will output the corresponding data classification results. The server labels each piece of data with the corresponding data type based on the classification results, dividing the data into two categories: crawler data and non-crawler data, and storing them separately. This completes the pre-classification of all website data, providing a data foundation for subsequent differentiated encryption protection. It eliminates the need to encrypt all website data, thus controlling computational overhead while ensuring security.

[0049] For example, the specific method for obtaining encrypted crawler data and returning non-crawler data and encrypted crawler data to the client can be as follows: When the server receives the first access request from the client, it first queries the corresponding target dataset based on the requested page information, filters out the content with crawler data tags from the dataset, performs symmetric encryption on this part of plaintext data, and converts it into ciphertext crawler data. Non-crawler data retains its original plaintext form. Finally, the result, which is a mixture of plaintext non-crawler data and ciphertext crawler data, is returned to the client. This allows non-core data to be displayed normally on the client's page, while core crawler data is presented in ciphertext form. Even if the crawler successfully initiates a request, it cannot directly obtain valid plaintext data.

[0050] For example, the specific method for access code generation and authentication triple construction can be as follows: When the server receives a second access request from the client for encrypted data, it first extracts the encrypted identifier and user identity information carried in the request, generates a dynamic access code uniquely bound to the encrypted data and the user, and sends the access code to the user's terminal device through a communication channel independent of the webpage. At the same time, the server binds and combines the encrypted identifier, user identifier, and generated access code locally to form a complete authentication triplet and stores it. The entire process of generating and storing the access code is completed on the server side. The frontend can only trigger the request operation and cannot obtain the generation logic and content of the access code, thus preventing the verification credentials from being reverse-engineered by the frontend from the root.

[0051] For example, the detection and decryption processing can be specifically handled as follows: After receiving a third access request, the server first extracts the user identifier and access code carried in the request as verification information. This verification information is then compared with the authentication triplet stored locally on the server. If the two match completely, the request is deemed legitimate. At this point, the server retrieves the decryption key and decryption algorithm corresponding to the ciphertext identifier from its storage database and returns them to the client. Upon receiving this, the client calls the decryption algorithm and inputs the decryption key to decrypt the local ciphertext crawler data, restoring the original plaintext crawler data and displaying it to the user. The entire decryption credential is only issued after verification and is bound to a single user and a single ciphertext, significantly increasing the difficulty of cracking.

[0052] For example, taking the anti-crawler protection of a sports news website as an example, the website backend first collects access records of all news data over the past three months, calculates the proportion of times each news item was accessed by a crawler tool to the total number of accesses, and labels the event details data with a proportion exceeding 70% as crawler data, and the homepage navigation and announcement data with a proportion below 30% as non-crawler data. This labeled data is used to train a crawler data recognition model, which is then used to classify and label all online news on the website, storing the corresponding tags. When a user opens a sports news page using a computer browser, the browser sends an initial access request to the website server, requesting the loading of the page's news list data. Upon receiving the request, the server queries the corresponding news dataset, filters out the event details content with crawler data tags, symmetrically encrypts the text values ​​of this content, retains the plaintext title keys, converts the body text into ciphertext, and generates a corresponding ciphertext identifier for each ciphertext item. Finally, the plaintext navigation and announcement data, along with the ciphertext-identified event data, are returned to the browser. After the browser renders the page, the navigation and announcements display correctly, but the event details are displayed as garbled text, with the "Request to View" button and phone number input box appearing in the garbled text locations. When a user wants to view the event details, they enter their phone number in the input box and click the "Request" button. The browser then sends a second access request to the server, carrying the encrypted identifier and the user's phone number. Upon receiving the second access request, the server generates a 6-digit dynamic access code and sends it to the user's mobile terminal via SMS. Simultaneously, the server stores the encrypted identifier, the user's phone number, and the generated access code locally as an authentication triple. After receiving the access code in the SMS message, the user enters the string of numbers in the access code input box on the webpage. The browser then sends a third access request to the server, carrying the phone number and the entered access code. After receiving a third access request, the server extracts the mobile phone number and access code from the request and compares them with the authentication triplet stored locally. Once the two are confirmed to be completely identical, the server retrieves the decryption key and decryption algorithm corresponding to the ciphertext identifier from the storage database and returns them to the user's browser. After receiving the decryption request, the browser calls the decryption algorithm and inputs the decryption key to decrypt the locally stored ciphertext data of the event details, restores the plaintext event details content, and displays it on the page for the user to browse normally.

[0053] As can be seen, in this embodiment, a pre-trained crawler data recognition model is used to classify and label historical website access data, distinguishing between crawler data and non-crawler data. Encryption is applied only to core data frequently accessed by crawlers. This achieves precise protection while converting the core target data of crawlers into ciphertext. When responding to the first access request, only plaintext non-crawler data and ciphertext crawler data are returned to the client. No decryption keys, decryption algorithms, or other core decryption credentials are issued throughout the process. This ensures that even if a crawler reverse-engineers the JavaScript code of the front-end page, it cannot obtain valid information for directly decrypting data from the front-end logic and returned data of the first interaction. This avoids the problem in traditional anti-crawler schemes where the token generation logic is exposed on the front-end and can be reverse-engineered. The core authentication credentials are completely isolated from the webpage front-end link. When responding to the second access request, the server generates a dynamic access code that is uniquely bound to the corresponding ciphertext crawler data and the requesting user. The access code is then sent to an independent user terminal. This is done only locally on the server. An authentication triplet consisting of a ciphertext identifier, a user identifier, and an access code is constructed. The generation logic and storage of the access code are controlled by the server, and the distribution channel is separated from the code execution environment of the web frontend. Crawlers cannot intercept, deduce, or simulate the generation of this access code by reverse engineering the frontend code. This solves the defect of single-backend protection, where the verification credential relies entirely on frontend interaction and is easily reverse engineered. The distribution of the core decryption credential is postponed to after the completion of multi-step verification. The corresponding decryption key and decryption algorithm are only distributed to the client after responding to the third access request and the authentication triplet is detected. The acquisition of decryption capability requires the completion of multi-step identity verification across terminals, and the decryption credential is strongly bound to the specific ciphertext and the specific user. Crawlers cannot directly obtain data decryption permissions by bypassing frontend verification or forging a single request. Compared with a single-backend protection scheme that only relies on backend token verification, this significantly increases the technical threshold and implementation cost for crawler reverse engineering, and effectively makes up for the security defects of single-backend protection, where the frontend logic is easily reverse engineered and the verification credential is easily replicated.

[0054] Preferably, in this embodiment, the acquisition of historical website access data involves classifying and labeling the historical access data based on a pre-trained crawler data recognition model to obtain crawler data and non-crawler data, including: The percentage of times each training data point was accessed by the crawler tool was calculated out of the total number of accesses, thus obtaining the crawler access percentage value corresponding to each training data point. The crawler access ratio of each access training data is compared with a preset ratio threshold. The access training data with a ratio greater than the preset ratio threshold is labeled with crawler data labels, and the access training data with a ratio less than or equal to the preset ratio threshold is labeled with non-crawler data labels, thus obtaining an access training dataset with binary labels. Based on the access training dataset with binary labels, the pre-built crawler data recognition model is trained under supervision to obtain the trained crawler data recognition model. The trained web crawler data recognition model is used to predict and classify the historical access data and label it accordingly, thus obtaining the web crawler data and the non-web crawler data.

[0055] Among them, the access training data is the historical access sample data of the website used to train the crawler data recognition model. Each access training data corresponds to a specific content data in the website. The backend will record the total number of accesses for each content data, as well as the number of accesses from the crawler tool. It has complete access source statistical attributes, which can support subsequent crawler access ratio statistics and tag labeling. It is the basic data for building model training samples.

[0056] Among them, the crawler access ratio is the proportion of times a single piece of training data is accessed by a crawler tool to the total number of times that data is accessed. This value can quantitatively reflect the degree to which the corresponding website data is of interest to crawler tools. The higher the value, the more likely the data is to become a crawler target. It is the core quantitative indicator for judging whether data belongs to crawler data.

[0057] The preset ratio threshold is a pre-set critical value used to distinguish between crawled data and non-crawled data. This threshold can be flexibly adjusted according to the website's protection requirements, data attributes, and business scenarios. When the crawled access ratio of a single piece of data is higher than this threshold, it is determined to be crawled data; when it is lower than or equal to this threshold, it is determined to be non-crawled data. This is a unified judgment benchmark for data classification and labeling.

[0058] Among them, binary labels are two mutually exclusive labels used to mark data categories, specifically crawler data labels and non-crawler data labels. Each access to training data will only be labeled with one of the two labels, without any intermediate transition state. This can clearly distinguish the category attributes of the data and adapt to the format requirements for sample labeling in supervised classification tasks.

[0059] The access training dataset is a sample set composed of all access training data that have been labeled. Each data item in the set has a corresponding binary label and includes a sufficient number of crawler data samples and non-crawler data samples as input materials for supervised training, providing complete sample support for parameter iteration and feature learning of the crawler data recognition model.

[0060] Supervised training is the mainstream training method for machine learning classification models. It takes a labeled access training dataset as input, uses the content attributes and access attributes of the data as input features of the model, and uses the corresponding binary labels as the expected output results. By iteratively adjusting the internal operation parameters of the model, the error between the model's prediction results and the true labels is reduced, allowing the model to gradually learn the feature differences between the two types of data, and finally has the ability to classify and predict unknown data.

[0061] For example, the specific method to obtain the crawler access ratio can be: The backend retrieves all access log records corresponding to all access training data, counts the cumulative number of times each data point was accessed by the crawler tool, and calculates the total number of accesses from all access sources for that data point. The crawler access count is divided by the total number of accesses to calculate the crawler access ratio for each data point, providing a quantitative basis for subsequent labeling.

[0062] For example, the specific method for binary label annotation processing can be as follows: After setting a pre-defined percentage threshold for classification, the percentage of crawler access corresponding to each piece of training data is compared with the threshold one by one. If the percentage is greater than the preset percentage threshold, the data is labeled as crawler data; if the percentage is less than or equal to the preset percentage threshold, the data is labeled as non-crawler data. Finally, all training data are labeled with binary labels, forming standardized classification training samples.

[0063] For example, the specific methods for supervised model training can be: Using a binary-labeled access training dataset as training material, the content features, access time period, access frequency, and other attributes of each data point are input into the initially constructed classification model. The corresponding binary label of the data is used as the target output. Through multiple iterations of training, the internal operation parameters of the model are continuously adjusted to reduce the deviation between the model classification result and the true label. When the classification accuracy of the model reaches the preset standard, the training process is completed, and a usable web crawler data recognition model is obtained.

[0064] For example, the specific methods for data prediction and classification processing can be as follows: The historical website access data to be processed is input one by one into the trained crawler data recognition model. Based on the classification feature rules learned during the training process, the model performs calculations on the input data and outputs the corresponding category prediction results. The server labels each historical access data with the corresponding category label according to the model output results, and finally obtains a complete set of crawler data and non-crawler data.

[0065] In this embodiment, the percentage of crawler visits is first statistically analyzed, and then training samples are constructed by comparing binary labels with thresholds. Using objective access data as the basis for labeling avoids subjective errors in manual labeling and ensures the accuracy and consistency of training sample labeling. Then, a crawler data recognition model is trained through supervised training, enabling the model to accurately learn the characteristic patterns of crawler data. Finally, the trained model automatically predicts and classifies historical access data, which can stably and efficiently complete the classification and labeling of large amounts of website data. This not only improves the accuracy of distinguishing between crawler and non-crawler data, making subsequent encryption protection more targeted and avoiding non-core data being additionally encrypted and affecting the browsing experience of normal users, but also reduces the operational costs of manual classification and provides a reliable data classification foundation for subsequent differentiated anti-crawler protection.

[0066] Preferably, in this embodiment, the step of encrypting the crawler data to obtain ciphertext crawler data includes: The crawler data is parsed to obtain structured data with key-value pairs where key names and value contents correspond one-to-one. The key names in the structured data are kept in plaintext without processing. Symmetric encryption is performed on the numerical content corresponding to each key name in the structured data to obtain the encrypted numerical content. The plaintext key name and the encrypted numerical content are combined according to a preset data structure to obtain the encrypted crawler data.

[0067] Key-value pair structure is a commonly used standardized data organization form for website interface data. It consists of paired field names and corresponding data content. Each field name uniquely corresponds to a data content. Multi-level nested structures can be set according to business needs, which can clearly define the field boundaries and content levels of the data, making it easy for the program to quickly locate and read the content of a specified field, and adapting to the data parsing and page rendering needs of the web front end.

[0068] Structured data refers to crawler data with a standardized key-value pair structure after format parsing. Each item in the data is identified by a corresponding field name, with clear field boundaries and a well-defined hierarchical relationship. This allows direct location of the field containing the core business content, facilitating subsequent targeted encryption processing of the core content without requiring full encryption of the entire data.

[0069] Among them, the key name is the name part in the key-value pair structure used to identify the field attribute. It is used to distinguish different types of data content, such as the field names of serial number, display title, and details in information data. It belongs to the structural identification information of the data, does not include the core business value content, and is the core basis for the front-end to parse the data structure and render the page layout framework.

[0070] Among them, the numerical content is the specific business data content corresponding to the key name in the key-value pair structure. It is a core value component of the website data, such as specific event text content, user data statistics results, etc. It is also the core target of crawling tools and belongs to the core data objects that need to be encrypted and protected in anti-crawler protection. After encryption, the effective information cannot be directly read.

[0071] Symmetric encryption is an encryption method that uses the same key for both encryption and decryption. It is fast and efficient in data processing, making it suitable for encrypting large amounts of text-based website data. Common encryption types include Advanced Encryption Standard (AES) and Triple Encryption Standard (DES). It can ensure encryption strength while also taking into account server response performance and concurrent processing capabilities.

[0072] The preset data structure is a pre-defined data organization format and hierarchical rules that are completely consistent with the original crawler data. It is used to guide the data reconstruction work after encryption, ensuring that the overall structure and field arrangement order of the encrypted data are consistent with the original data. This allows the front end to use the original parsing logic to process the data without the need for additional code adaptation and modification for the encrypted data.

[0073] For example, the specific methods for obtaining structured data can be: The server reads the complete content of the raw crawler data, identifies the data organization format and nesting hierarchy rules, and decomposes the serialized or multi-layered nested raw data into a clear key-value pair format. It separates the key name and value content corresponding to each field, so that the data structure identification part and the core business content part are clearly separated, providing an independent processing unit that can be precisely operated for subsequent selective encryption.

[0074] For example, selective encryption processing can specifically include: The server iterates through all the parsed key-value pairs, performs no encryption on the key names of all fields, and keeps them in their original plaintext state. At the same time, it calls a symmetric encryption algorithm and passes in a pre-generated encryption key to perform encryption operations on the numerical content corresponding to each key name, transforming the plaintext business content into ciphertext content that cannot be directly read. This achieves a differentiated encryption effect that encrypts only the core business data while retaining the data structure identifier.

[0075] For example, the specific methods of data combination processing can be as follows: The server will retain all keys in plaintext and recombine them with the corresponding encrypted ciphertext values ​​according to the hierarchical relationship and field arrangement of the original data, restoring them to complete data that is completely consistent with the original data format. This results in ciphertext crawler data that is structurally compatible and has encrypted content, ensuring the compatibility of front-end data parsing.

[0076] In this embodiment, key-value pair structured data is first parsed, and then a selective encryption method is adopted, which preserves the plaintext of the keys and encrypts the numerical content. On the one hand, this precisely protects the core business content that the crawler is interested in, ensuring that even if the crawler successfully obtains the interface response data, it cannot extract valid business information, thus guaranteeing data security at the content level. On the other hand, the structural identifier of the data is preserved, allowing the front-end to normally parse the data structure and render the basic page layout without affecting the basic page browsing experience of normal users due to encryption operations. At the same time, encrypting only the core numerical content reduces the amount of data processing for encryption operations. Combined with the efficient processing characteristics of symmetric encryption operations, it effectively reduces the server performance overhead of the encryption process and can adapt to high-concurrency access scenarios of the website. In addition, the encrypted data is reconstructed according to the preset data structure, ensuring high compatibility of the data format before and after encryption. No large-scale modification of the front-end data parsing logic is required, which greatly reduces the implementation cost of the solution.

[0077] Preferably, in the embodiment, the construction of the authentication triplet including the ciphertext identifier, the user identifier, and the access code includes: The encrypted values ​​corresponding to each key in a single encrypted crawler data are concatenated in a preset order to obtain a continuous string. The continuous string is then subjected to one-way message digest encoding to obtain a ciphertext identifier that corresponds one-to-one with the encrypted crawler data. Extract the user's mobile phone number carried in the second access request, and use the user's mobile phone number as the user identifier of the authentication triplet; The ciphertext identifier, the user identifier, and the access code are combined to obtain the authentication triplet.

[0078] The preset order is the pre-defined concatenation order of each encrypted numerical field in a single piece of encrypted crawler data. This order is consistent with the field arrangement order of the original data, ensuring that the string obtained by concatenating the same piece of encrypted data is exactly the same each time. The concatenation results of different pieces of encrypted data are different. This is the basic rule for generating unique encrypted identifiers, ensuring the consistency and reproducibility of encrypted identifier generation.

[0079] Among them, the continuous string is a whole unseparated character sequence formed by concatenating all the encrypted numerical content in a single ciphertext crawler data in a preset order. It completely includes the character information of all the encrypted numerical values ​​in the ciphertext data and serves as the input material for one-way message digest encoding. The uniqueness of its content determines the uniqueness of the final generated ciphertext identifier.

[0080] One-way message digest encoding is an encoding calculation method that converts an input character sequence of arbitrary length into an output character sequence of fixed length. The operation process has the characteristics of being one-way and irreversible, and it is impossible to deduce the original input content from the output result. Moreover, the output results corresponding to different input contents are almost never repeated. It can generate a unique identity for each piece of ciphertext data, while not revealing the original content information of the ciphertext.

[0081] The user identifier is information used to uniquely identify the user who initiates the access request. In this solution, the user's mobile phone number is used as the user identifier, which is unique and verifiable. It can bind the decryption request to a specific user, prevent anonymous identities from arbitrarily applying for data decryption permissions, and support subsequent cross-terminal identity verification.

[0082] Among them, the user's mobile phone number is the mobile communication number held by the user. It is actively entered by the user when requesting to view the encrypted data and sent to the server along with the second access request. It serves as a unique identifier for identifying the user and also as a channel for receiving and distributing the access code, thereby binding the user's identity with the receiving terminal and improving the reliability of identity verification.

[0083] For example, the specific method for obtaining the ciphertext identifier can be: First, determine the order of the fields corresponding to each encrypted value in a single piece of encrypted crawler data. Then, concatenate all the encrypted values ​​in this fixed order to form a continuous string. Next, input this string into a one-way message digest algorithm for encoding. Finally, obtain a fixed-length encoded result as the ciphertext identifier corresponding to this piece of encrypted crawler data. Each piece of encrypted data corresponds to a unique ciphertext identifier, which can be used as the identity index of the encrypted data.

[0084] For example, the user identifier extraction process can be specifically handled as follows: After receiving the second access request from the client, the server parses the parameters of the request message, extracts the mobile phone number information submitted by the user, and uses the mobile phone number as the identity identifier of the corresponding user to associate the user's decryption application behavior with the access code distribution object, thereby realizing the binding of the application behavior with the individual user.

[0085] For example, the specific method for processing authentication triplet combinations can be as follows: The server binds and combines the generated ciphertext identifier corresponding to the current ciphertext data, the extracted user identifier corresponding to the user, and the dynamic access code generated for this application to form a complete set of authentication triplet data, which is stored in the local storage medium of the server as the basis for identity verification of subsequent third access requests. The complete content and combination rules of the triplet will not be exposed to the front end throughout the process.

[0086] In this embodiment, a ciphertext identifier is generated by concatenating ciphertext values ​​and then performing one-way message digest encoding. This ensures that each piece of ciphertext crawler data has a unique identifier, accurately corresponding to specific ciphertext data and decryption credentials. Furthermore, the irreversible nature of one-way encoding prevents the ciphertext identifier from leaking the original content information of the ciphertext. The fixed identifier length facilitates rapid indexing and querying by the server. Using the user's mobile phone number as the user identifier achieves unique binding of the user's identity and unifies the receiving terminal of the identity identifier and access code, ensuring that the access code is accurately delivered to the real user and preventing crawlers from impersonating anonymous individuals to request decryption permissions. Finally, the ciphertext identifier, user identifier, and dynamic access code are combined into an authentication triple, with each three bound together. This makes the verification dimensions for each decryption request more comprehensive, verifying both the user's identity and the matching relationship between the access code and the corresponding ciphertext. This avoids the problem of single-dimensional verification being easily bypassed, effectively improving the accuracy and security of identity verification and providing a reliable verification foundation for the subsequent issuance of decryption credentials.

[0087] Preferably, in an embodiment, the method further includes: In response to the first access request initiated by the client, the non-crawler data and the encrypted crawler data are returned to the client, without issuing any decryption credentials to the client; In response to the second access request initiated by the client based on the encrypted crawler data, an access code is generated and sent to the user terminal, but the decryption key and decryption algorithm are not sent. In response to the third access request initiated by the client, after completing the verification information detection, the decryption key and decryption algorithm are sent to the client.

[0088] Among them, the decryption certificate is a collective term for all the core information that can restore the encrypted crawler data to the plaintext crawler data. It mainly includes two types of content: decryption key and decryption algorithm. It is a necessary condition for completing the decryption of encrypted data and has extremely high security sensitivity. Once illegally obtained, the corresponding encrypted data can be directly cracked. Therefore, it is necessary to strictly control the timing of issuance and the access permissions, and issue it only to legitimate users who have passed the full identity verification.

[0089] For example, the initial request for data return processing can be specifically handled as follows: After receiving the first access request from the client, the server only organizes the plaintext non-crawler data and the ciphertext crawler data into a response result and returns it to the client. The response content does not include any form of decryption credentials, and neither the decryption key nor the decryption algorithm is returned. This ensures that the first interaction only completes the distribution of basic page data and does not expose any core information that can be used for decryption, thus preventing the crawler from directly obtaining decryption capabilities by simulating basic data requests.

[0090] For example, the specific method for issuing a secondary request access code can be as follows: After receiving the second access request initiated by the client based on encrypted data, the server only generates a dynamic access code corresponding to this application and sends it to the user terminal through an independent channel. The response returned to the web client only includes the application acceptance status and does not send the decryption key or decryption algorithm. The access code is only used as an intermediate credential for identity verification and does not have the ability to decrypt data itself. Even if it is illegally intercepted, the encrypted content cannot be directly restored.

[0091] For example, the specific method for processing the issuance of decryption credentials through three requests can be as follows: After receiving a third access request from a client, the server prioritizes comparing and verifying the information. Only after confirming that the requester is a legitimate user will the server return the decryption key and decryption algorithm that match the corresponding ciphertext to the client. The issuance of the core decryption credential is postponed until after identity verification is completed, and the conditions and scope of the decryption credential issuance are strictly limited.

[0092] In this embodiment, a phased and tiered credential distribution mechanism completely isolates the core decryption credential from the basic data requests and verification application requests in terms of timing. The first request only distributes the data itself, the second request only distributes the intermediate verification credential, and the decryption key and decryption algorithm are only distributed in the third stage after the complete identity verification is completed. This completely avoids the problem of the decryption credential and data being returned synchronously in traditional solutions, which is easily intercepted by a single interface request. This makes it impossible for crawlers to obtain complete decryption capabilities by reverse engineering the front-end code or simulating a single interface request, greatly improving the security of the core decryption credential. At the same time, the layered distribution mechanism ensures that each interaction only exposes information at the corresponding level, effectively reducing the exposure scope of sensitive information. From the perspective of credential control, this strengthens the overall anti-crawler system's resistance to reverse engineering and further compensates for the security defects of easy credential leakage and single protection level in single back-end protection.

[0093] Preferably, in an embodiment, detecting the verification information carried in the third access request based on the authentication triplet includes: Extract the user identifier and access code from the authentication triplet as the first verification information; Extract the user identifier and access code from the third access request as the second verification information; The first verification information is compared with the second verification information to obtain the information comparison result; If the information comparison results show that the information matches, the detection result is determined to be passed.

[0094] The first verification information is a combination of user identifier and access code extracted from the authentication triplet stored locally on the server. It is the original legal verification benchmark retained by the server and is generated and stored synchronously when the user initiates the second access request. It is not exposed to the outside world at all and is not affected by the front-end interaction logic. It has the characteristic of being tamper-proof and is the reference standard in the identity verification process. It is used to determine whether the verification information submitted by the client is legal and valid.

[0095] The second verification information is a combination of user identifier and access code extracted from the parameters of the third access request initiated by the client. It is a verification credential submitted by the client side, which is uploaded along with the request after the user enters relevant information on the web page. It represents the identity and verification credentials claimed by the current request initiator. Its authenticity and legality can only be confirmed after comparison with the server's baseline information. It is the verification object for identity verification.

[0096] The information comparison result is a verification conclusion obtained by matching and verifying the first verification information with the second verification information item by item. It is divided into two states: information consistency and information inconsistency. It is the direct basis for deciding whether to issue a decryption certificate. Information consistency means that the verification information submitted by the client completely matches the legitimate certificate stored by the server, and the request is legitimate. Information inconsistency means that the verification information is incorrect or forged, and the request does not have decryption authority.

[0097] For example, the specific method for extracting and processing benchmark verification information can be as follows: After receiving a third access request, the server first locates the authentication triplet of the corresponding user and the corresponding encrypted text based on the associated information carried in the request. It then extracts the pre-stored user identifier and access code from the triplet to form the first verification information. This information comes from the raw data stored locally on the server and has not gone through the front-end transmission process, which can ensure the authenticity and accuracy of the verification benchmark and prevent the verification standard from being tampered with.

[0098] For example, the specific method for extracting and processing the verification information to be verified can be as follows: The server parses the message content of the third access request and extracts the user identifier and access code submitted by the user from the form parameters or interface parameters carried in the request to form the second verification information. This information is uploaded by the client and may be subject to input errors, forgery, or tampering. Therefore, it needs to be verified for its legitimacy by comparing it with the baseline information.

[0099] For example, the information comparison processing method can specifically be as follows: The server performs item-by-item matching and verification between the first verification information and the second verification information. First, it compares whether the user identifiers in the two sets of information are completely identical, and then it compares whether the access codes in the two sets of information are completely identical. Only when both items are completely matched is the information determined to be consistent. If either item does not match, the information is determined to be inconsistent. Finally, the corresponding information comparison result is generated, completing the core judgment process of verification.

[0100] For example, the specific method for determining the detection result can be: The server makes a final permission determination based on the results of the information comparison process. When the information comparison results show that the information matches, the server determines that the detection result of this third access request is passed, and the requester has the legitimate permission to obtain the decryption certificate. When the information comparison results show that the information does not match, the server determines that the detection result is failed, and the server refuses to issue the decryption certificate for this request, thereby achieving precise control over decryption permissions.

[0101] In this embodiment, a verification logic that compares server-side baseline information and client-side verification information is adopted. The authentication triplet stored locally is used as the verification baseline, and the verification standard is not exposed to the outside world throughout the process. This fundamentally avoids the risk of verification rules being reverse-engineered by the front end and verification baseline being tampered with. At the same time, the user identifier and access code are double-verified, which verifies both the legitimacy of the user's identity and the binding relationship between the access code and the user. This prevents the vulnerability of bypassing verification by the misuse of a single piece of information, effectively improving the anti-forgery capability and accuracy of identity verification. In addition, the item-by-item comparison logic has high computational efficiency and can quickly complete the verification in high-concurrency scenarios. While ensuring the strength of security protection, it also takes into account the user's access experience, providing a reliable judgment basis for the secure issuance of decryption credentials, and further strengthening the overall anti-crawler system's protection reliability.

[0102] Preferably, in this embodiment, the step of encrypting the crawler data to obtain ciphertext crawler data includes: For business scenarios where the data security level is higher than a preset level threshold and the concurrency is lower than a preset concurrency threshold, the crawler data is encrypted in real time after receiving the first access request to generate the encrypted crawler data. For business scenarios where the concurrency exceeds a preset concurrency threshold or the access volume exceeds a preset access volume threshold, the crawler data is encrypted before the first access request is received, generating the ciphertext crawler data and storing the ciphertext crawler data in the ciphertext database.

[0103] Data security level is a standard that classifies website data based on its business value, the impact of a leak, and the sensitivity of the content. The higher the level, the higher the core value of the data and the greater the loss caused by illegal scraping. The corresponding protection requirements are also higher. It is one of the core criteria for selecting encryption processing strategies and is used to match encryption protection schemes of different strengths.

[0104] Among them, the preset level threshold is a pre-set critical value for data security level, which is used to classify business scenarios with different security requirements. When the data security level of a business scenario is higher than this threshold, it belongs to a high security requirement scenario and requires the use of encryption processing methods with higher security strength. It is the benchmark for judging the security dimension of scenario classification and can be flexibly adjusted according to business protection requirements.

[0105] Concurrency refers to the number of access requests that a server receives and processes simultaneously within a unit of time. It is a core indicator for measuring server load. Higher concurrency means that the server needs to process more requests within the same time period, resulting in greater computational pressure. It is an important performance dimension for choosing the timing of encryption and directly affects the degree to which encryption operations impact system response speed.

[0106] The preset concurrency threshold is a pre-defined critical value for concurrency, used to distinguish between low-concurrency and high-concurrency business scenarios. When the concurrency of a business scenario is lower than this threshold, the server has sufficient computing pressure to handle encrypted operations synchronized with requests; when it is higher than this threshold, pre-encryption is required to reduce real-time computing pressure and ensure system response efficiency.

[0107] Real-time encryption processing involves synchronously performing encryption operations on the target crawler data and generating ciphertext crawler data after receiving the first access request from the client. The encryption action corresponds one-to-one with the user request and is triggered synchronously. The key and encryption rules can be flexibly adjusted for each encryption, resulting in stronger encryption dynamism and security, and making it suitable for business scenarios with low concurrency and high security requirements.

[0108] Among them, the number of visits refers to the total number of access requests received in a certain business scenario within a fixed statistical period. It reflects the popularity and access frequency of the corresponding data. The higher the number of visits, the greater the total overhead of repeatedly performing encryption operations. It is an important reference indicator for judging whether to use pre-encryption and is used to evaluate the performance loss of repeated encryption.

[0109] The preset access threshold is a pre-defined critical value for access volume, used to distinguish between high-frequency and low-frequency access scenarios. When the access volume of a business scenario exceeds this threshold, the data is requested frequently, making it suitable for pre-encryption to reduce redundant calculations. When the access volume is below this threshold, the overhead of repeated encryption is controllable, and real-time encryption can be used to ensure security. It serves as the benchmark for judging the access frequency dimension of scenario classification.

[0110] The ciphertext database is an independent storage unit used to store pre-generated ciphertext crawler data. Each piece of ciphertext data corresponds to a unique ciphertext identifier, which is associated one-to-one with the identifier of the plaintext data. When an access request is received, the corresponding ciphertext can be retrieved quickly through the identifier without performing encryption operations in real time. This can effectively reduce the server load during the request and response phase and support stable operation in high-concurrency scenarios.

[0111] For example, a specific method for real-time encryption processing in high-security, low-concurrency scenarios can be: The server does not encrypt the crawler data in advance when it does not receive an access request. When it receives the first access request from the client, it immediately calls the encryption algorithm to perform encryption operation on the crawler data corresponding to the request, and returns the encrypted crawler data to the client along with the response. This processing method has an independent encryption process for each request, and the encryption key and encryption rules can be dynamically configured, which can achieve higher security protection without occupying storage resources in advance, and is suitable for business scenarios with higher security priority.

[0112] For example, the pre-encrypted storage processing method for high-concurrency, high-access scenarios can specifically be as follows: During off-peak hours, the server pre-encrypts all crawler data for the scenario, generating corresponding ciphertext crawler data and assigning a unique ciphertext identifier to each ciphertext. All ciphertext data is then stored in a ciphertext database. When the server receives the first access request from the client, it does not need to perform encryption. Instead, it retrieves the pre-generated ciphertext crawler data from the ciphertext database based on the ciphertext identifier, replaces the plaintext, and returns it to the client. This shifts the pressure of encryption from the request-response phase to idle periods, effectively reducing the real-time computing load in high-concurrency scenarios.

[0113] In this embodiment, business scenarios are divided based on three dimensions: data security level, concurrency, and access volume. Two different encryption processing times—real-time encryption and pre-encrypted storage—are specifically matched. For high-security, low-concurrency scenarios, request-triggered real-time encryption is used, allowing for the configuration of independent encryption parameters for each request. This enhances the dynamism of encryption and the difficulty of cracking, maximizing the security of high-value data. For high-concurrency, high-access scenarios, a pre-encrypted storage solution is employed, shifting the encryption computation load from the request-response phase to idle business periods. This avoids encryption computation slowing down system response speed under high concurrency, effectively improving the system's concurrency capacity and access stability. The two encryption strategies can be flexibly selected and switched according to the actual needs of different businesses, balancing different levels of data security protection requirements with the trade-off between security protection and system performance. This avoids the limitation of a single encryption mode being unable to adapt to diverse business scenarios, effectively improving the scenario adaptability and practicality of the entire anti-crawler solution.

[0114] To better implement the above methods, this application also provides an anti-crawler device, which can be integrated into electronic devices, such as servers or terminals. The terminal may include tablet computers, laptop computers, and / or personal computers.

[0115] For example, see Figure 2 , Figure 2 This is a schematic diagram of the anti-crawler device provided in an embodiment of the present invention, as shown below. Figure 2 As shown, the anti-crawler device may include an acquisition module 201, an encryption module 202, a construction module 203, and a detection module 204, as follows: The acquisition module 201 is used to acquire historical access data of the website. Based on a pre-trained crawler data recognition model, the historical access data is classified and labeled to obtain crawler data and non-crawler data. The crawler data is website data accessed by crawler tools, and the non-crawler data is website data accessed by normal web page clients. Encryption module 202 is used to encrypt the crawler data in response to the first access request initiated by the client, obtain ciphertext crawler data, and return the non-crawler data and the ciphertext crawler data to the client; The construction module 203 is used to respond to the second access request initiated by the client based on the encrypted crawler data, generate an access code corresponding to the encrypted crawler data and the requesting user and send it to the user terminal, and construct an authentication triplet including the encrypted identifier, the user identifier and the access code; The detection module 204 is used to respond to the third access request initiated by the client, and to detect the verification information carried in the third access request according to the authentication triplet. If the detection result is successful, the module sends a decryption key and a decryption algorithm to the client so that the client can decrypt the encrypted crawler data according to the decryption key and the decryption algorithm to obtain the plaintext crawler data.

[0116] In practice, the above modules can be implemented as independent entities or combined in any way to be implemented as the same or several entities. For the specific implementation of the above modules, please refer to the previous method implementation examples, which will not be repeated here.

[0117] As can be seen from the above, in this embodiment, a pre-trained crawler data recognition model is used to classify and label historical website access data, distinguishing between crawler data and non-crawler data. Encryption is applied only to core data frequently accessed by crawlers. This achieves precise protection while converting the core target data of the crawler into ciphertext. Furthermore, in response to the first access request, only plaintext non-crawler data and ciphertext crawler data are returned to the client. No decryption keys, decryption algorithms, or other core decryption credentials are issued throughout the process. This ensures that even if the crawler reverse-engineers the JavaScript code of the front-end page, it cannot obtain valid information for directly decrypting data from the front-end logic and returned data of the initial interaction. This avoids the problem in traditional anti-crawler schemes where the token generation logic is exposed on the front-end and can be reverse-engineered. The core authentication credentials are completely isolated from the webpage front-end link. In response to the second access request, the server generates a dynamic access code that is uniquely bound to the corresponding ciphertext crawler data and the requesting user, and sends the access code to an independent user terminal. Simultaneously, this is only performed on the server. The client-side constructs an authentication triplet containing a ciphertext identifier, a user identifier, and an access code. The generation logic and storage of the access code are controlled by the server, and the distribution channel is independent of the web front-end code execution environment. Crawlers cannot intercept, deduce, or simulate the generation of this access code by reverse engineering the front-end code. This solves the defect of single-backend protection, where the verification credential relies entirely on front-end interaction and is easily reverse engineered. The distribution of the core decryption credential is postponed to after multi-step verification is completed. Only after responding to the third access request and the authentication triplet is detected and passed, is the corresponding decryption key and decryption algorithm distributed to the client. Obtaining decryption capability requires completing multi-step authentication across terminals, and the decryption credential is strongly bound to the specific ciphertext and the specific user. Crawlers cannot directly obtain data decryption permissions by bypassing front-end verification or forging a single request. Compared with single-backend protection solutions that only rely on back-end token verification, this significantly increases the technical threshold and implementation cost for crawler reverse engineering, effectively making up for the security defects of single-backend protection, where the front-end logic is easily reverse engineered and the verification credential is easily replicated.

[0118] In some embodiments, the anti-crawler device of the present invention can be implemented in a combination of hardware and software. As an example, the anti-crawler device of the present invention can be a processor in the form of a hardware decoding processor, which is programmed to execute the anti-crawler method of the present invention. For example, the processor in the form of a hardware decoding processor can be one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0119] The modules described in the embodiments of this invention can be implemented in software or hardware. The names of the modules are not, in some cases, limiting the scope of the module itself.

[0120] An electronic device according to an embodiment of the present invention includes a processor and a memory, wherein the memory stores multiple instructions; the processor loads instructions from the memory to execute the steps in any of the above-mentioned anti-crawler methods, that is, an electronic device according to an embodiment of the present invention may include, but is not limited to: a processor and a memory; the memory is used to store a computer program; the processor is used to execute the anti-crawler method shown in any embodiment of the present invention by calling the computer program.

[0121] In one alternative embodiment, an electronic device is provided, such as Figure 3 As shown, Figure 3 The illustrated electronic device 3000 includes a processor 3001 and a memory 3003. The processor 3001 and the memory 3003 are connected, for example, via a bus 3002. Optionally, the electronic device 3000 may further include a transceiver 3004, which can be used for data interaction between the electronic device and other electronic devices, such as sending and / or receiving data. It should be noted that in practical applications, the transceiver 3004 is not limited to one type, and the structure of the electronic device 3000 does not constitute a limitation on the embodiments of the present invention.

[0122] Processor 3001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this invention. Processor 3001 may also be a combination that implements computational functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0123] Bus 3002 may include a path for transmitting information between the aforementioned components. Bus 3002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. Bus 3002 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 3 The bus 3002 is represented by only one thick line, but this does not mean that there is only one bus or one type of bus.

[0124] The memory 3003 may be ROM (Read Only Memory) or other types of static storage devices capable of storing static information and instructions, RAM (Random Access Memory) or other types of dynamic storage devices capable of storing information and instructions, or EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read Only Memory) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto.

[0125] The memory 3003 stores application code (computer program) for executing the present invention, and its execution is controlled by the processor 3001. The processor 3001 executes the application code stored in the memory 3003 to implement the content shown in the foregoing method embodiments.

[0126] Among them, electronic devices can also be terminal devices, which can be any device that can install applications, including at least one of smartphones, tablets, laptops, desktop computers, smart speakers, smartwatches, smart TVs, and smart in-vehicle devices.

[0127] It should be noted that, Figure 3 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments of the present invention.

[0128] An embodiment of the present invention provides a computer-readable storage medium storing a plurality of instructions adapted for loading by a processor to execute the steps in any of the above-described anti-crawler methods.

[0129] Alternatively, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), magnetic tape, a floppy disk, and an optical data storage device, etc.

[0130] In an exemplary embodiment, a computer program product or computer program is also provided, which includes computer instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the electronic device to perform the steps in any of the aforementioned anti-crawler methods.

[0131] Computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0132] It should be understood that the flowcharts and block diagrams in the accompanying drawings illustrate the implementable architecture, functions, and operations of methods and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code, which includes one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0133] The computer-readable storage medium provided in this invention can be, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EEPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium that includes or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0134] The aforementioned computer-readable storage medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the method shown in the above embodiments.

[0135] The above description is merely a preferred embodiment of the present invention and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of disclosure in this invention is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in this invention.

[0136] It should be noted that the terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and represent a limitation on a specific order or sequence. Where appropriate, the order of use for similar objects can be interchanged so that the embodiments of this application described herein can be implemented in an order other than that shown or described.

[0137] Those skilled in the art will recognize that this invention can be implemented as a system, method, or computer program product. Therefore, this invention can be specifically implemented in the following forms: it can be entirely hardware, entirely software (including firmware, resident software, microcode, etc.), or a combination of hardware and software, generally referred to herein as a "circuit," "module," or "system." Furthermore, in some embodiments, this invention can also be implemented as a computer program product in one or more computer-readable media, the computer-readable media including computer-readable program code.

[0138] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. An anti-crawler method, characterized in that, include: Historical access data of a website is obtained, and the historical access data is classified and labeled based on a pre-trained crawler data recognition model to obtain crawler data and non-crawler data. The crawler data is website data accessed by crawler tools, and the non-crawler data is website data accessed by normal web page clients. In response to the first access request initiated by the client, the crawler data is encrypted to obtain ciphertext crawler data, and the non-crawler data and the ciphertext crawler data are returned to the client. In response to the second access request initiated by the client based on the encrypted crawler data, an access code corresponding to the encrypted crawler data and the requesting user is generated and sent to the user terminal, and an authentication triplet including the encrypted identifier, the user identifier and the access code is constructed. In response to a third access request initiated by the client, the authentication information carried in the third access request is checked according to the authentication triplet. If the check result is successful, a decryption key and decryption algorithm are sent to the client so that the client can decrypt the encrypted crawler data according to the decryption key and the decryption algorithm to obtain the plaintext crawler data.

2. The anti-crawler method according to claim 1, characterized in that, The process of obtaining historical website access data involves classifying and labeling the historical access data based on a pre-trained web crawler data recognition model to obtain crawler data and non-crawler data, including: The percentage of times each training data point was accessed by the crawler tool was calculated out of the total number of accesses, thus obtaining the crawler access percentage value corresponding to each training data point. The crawler access ratio of each access training data is compared with a preset ratio threshold. The access training data with a ratio greater than the preset ratio threshold is labeled with crawler data labels, and the access training data with a ratio less than or equal to the preset ratio threshold is labeled with non-crawler data labels, thus obtaining an access training dataset with binary labels. Based on the access training dataset with binary labels, the pre-built crawler data recognition model is trained under supervision to obtain the trained crawler data recognition model. The trained web crawler data recognition model is used to predict and classify the historical access data and label it accordingly, thus obtaining the web crawler data and the non-web crawler data.

3. The anti-crawler method according to claim 1, characterized in that, The step of encrypting the crawler data to obtain ciphertext crawler data includes: The crawler data is parsed to obtain structured data with key-value pairs where key names and value contents correspond one-to-one. The key names in the structured data are kept in plaintext without processing. Symmetric encryption is performed on the numerical content corresponding to each key name in the structured data to obtain the encrypted numerical content. The plaintext key name and the encrypted numerical content are combined according to a preset data structure to obtain the encrypted crawler data.

4. The anti-crawler method according to claim 1, characterized in that, The construction of the authentication triple, comprising a ciphertext identifier, a user identifier, and an access code, includes: The encrypted values ​​corresponding to each key in a single encrypted crawler data are concatenated in a preset order to obtain a continuous string. The continuous string is then subjected to one-way message digest encoding to obtain a ciphertext identifier that corresponds one-to-one with the encrypted crawler data. Extract the user's mobile phone number carried in the second access request, and use the user's mobile phone number as the user identifier of the authentication triplet; The ciphertext identifier, the user identifier, and the access code are combined to obtain the authentication triplet.

5. The anti-crawler method according to claim 1, characterized in that, The method further includes: In response to the first access request initiated by the client, the non-crawler data and the encrypted crawler data are returned to the client, without issuing any decryption credentials to the client; In response to the second access request initiated by the client based on the encrypted crawler data, an access code is generated and sent to the user terminal, but the decryption key and decryption algorithm are not sent. In response to the third access request initiated by the client, after completing the verification information detection, the decryption key and decryption algorithm are sent to the client.

6. The anti-crawler method according to claim 1, characterized in that, The step of detecting the verification information carried in the third access request based on the authentication triplet includes: Extract the user identifier and access code from the authentication triplet as the first verification information; Extract the user identifier and access code from the third access request as the second verification information; The first verification information is compared with the second verification information to obtain the information comparison result; If the information comparison results show that the information matches, the detection result is determined to be passed.

7. The anti-crawler method according to claim 1, characterized in that, The step of encrypting the crawler data to obtain ciphertext crawler data includes: For business scenarios where the data security level is higher than a preset level threshold and the concurrency is lower than a preset concurrency threshold, the crawler data is encrypted in real time after receiving the first access request to generate the encrypted crawler data. For business scenarios where the concurrency exceeds a preset concurrency threshold or the access volume exceeds a preset access volume threshold, the crawler data is encrypted before the first access request is received, generating the ciphertext crawler data and storing the ciphertext crawler data in the ciphertext database.

8. An anti-crawler device, characterized in that, include: The acquisition module is used to acquire historical access data of the website. Based on a pre-trained crawler data recognition model, the historical access data is classified and labeled to obtain crawler data and non-crawler data. The crawler data is website data accessed by crawler tools, and the non-crawler data is website data accessed by normal web page clients. An encryption module is used to encrypt the crawler data in response to a first access request initiated by the client, obtain ciphertext crawler data, and return the non-crawler data and the ciphertext crawler data to the client. The construction module is used to respond to the second access request initiated by the client based on the encrypted crawler data, generate an access code corresponding to the encrypted crawler data and the requesting user and send it to the user terminal, and construct an authentication triple including the encrypted identifier, the user identifier and the access code; The detection module is used to respond to a third access request initiated by the client, and to detect the verification information carried in the third access request based on the authentication triplet. If the detection result is successful, the module sends a decryption key and a decryption algorithm to the client so that the client can decrypt the encrypted crawler data according to the decryption key and the decryption algorithm to obtain the plaintext crawler data.

9. An electronic device, characterized in that, It includes a processor and a memory, the memory storing multiple instructions; the processor loads instructions from the memory to perform the steps in the anti-crawler method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a plurality of instructions adapted for loading by a processor to execute the steps of the anti-crawler method according to any one of claims 1 to 7.

Images