Anti-crawler method and device, electronic equipment and storage medium

Abstract

The invention discloses an anti-crawler method and device, electronic equipment and a storage medium, and when a server determines that a first webpage acquisition request sent by a client comprises an initial identifier, a first identifier generation algorithm is sent to the client. And the client executes the first identifier generation algorithm to obtain the first identifier, and sends a second webpage acquisition request containing the first identifier and the initial identifier to the server. And the server verifies the first identifier and the initial identifier in the second webpage acquisition request, and sends a second identifier generation algorithm to the client when the verification is passed. And the client executes a second identifier generation algorithm, obtains a second identifier based on the first identifier and the initial identifier, and sends a third webpage acquisition request containing the first identifier, the second identifier and the initial identifier to the server. And the server verifies the first identifier, the second identifier and the initial identifier in the third webpage acquisition request, and when the verification is passed, the server sends the webpage content requested by the client to the client.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to an anti-crawler method, device, electronic device and storage medium. Background Art

[0002] With the development of science and technology, Internet technology has developed rapidly. In the field of Internet information search, a crawler is a program or script that automatically captures Internet information according to certain rules.

[0003] However, frequent crawling of web content by crawlers will increase the load pressure of the server, and frequent crawling of web content by crawlers will also affect normal users' access to web content and affect user experience. Therefore, anti-crawling is needed, that is, formulating strategies to inhibit illegal or improper crawling of web content by crawlers.

[0004] Based on this, this specification provides an anti-crawler method. Summary of the invention

[0005] This specification provides an anti-crawler method, device, electronic device and storage medium to partially solve the above-mentioned problems existing in the prior art.

[0006] This manual adopts the following technical solutions:

[0007] This specification provides an anti-crawler method, which is applied to a server; the method comprises:

[0008] Receive a first webpage acquisition request sent by a client;

[0009] When it is determined that the first webpage acquisition request includes an initial identifier, sending a first identifier generation algorithm to the client, so that the client generates a first identifier according to the first identifier generation algorithm, and causing the client to send a second webpage acquisition request, wherein the second webpage acquisition request includes the initial identifier and the first identifier;

[0010] receiving the second webpage acquisition request, and when determining that the second webpage acquisition request includes the initial identifier and the first identifier, sending a second identifier generation algorithm to the client, so that the client generates a second identifier based on the initial identifier and the first identifier through the second identifier generation algorithm, and causing the client to send a third webpage acquisition request, wherein the third webpage acquisition request includes the initial identifier, the first identifier, and the second identifier;

[0011] The third webpage acquisition request is received, and when it is determined that the third webpage acquisition request includes the initial identifier, the first identifier, and the second identifier, the webpage content requested by the client is sent to the client.

[0012] Optionally, after receiving the first webpage acquisition request sent by the client and before receiving the second webpage acquisition request, the method further includes:

[0013] When it is determined that the first web page acquisition request does not include an initial identifier, an initial identifier is determined; and the initial identifier and the first identifier generation algorithm are sent to the client, so that the client generates a first identifier according to the first identifier generation algorithm, and the client sends a second web page acquisition request, wherein the second web page acquisition request includes the first identifier and the initial identifier.

[0014] Optionally, sending the first identifier generation algorithm to the client specifically includes:

[0015] Using code reinforcement technology to reinforce the first identifier generation algorithm to obtain a reinforced first identifier generation algorithm;

[0016] Sending the obtained reinforced first identifier generation algorithm to the client;

[0017] Sending the second identifier generation algorithm to the client specifically includes:

[0018] Using code reinforcement technology to reinforce the generation of the second identifier, and obtain a reinforced second identifier generation algorithm;

[0019] The obtained reinforced second identifier generation algorithm is sent to the client.

[0020] This specification provides an anti-crawler method, which is applied to a client; the method includes:

[0021] When it is determined that the initial identifier is obtained, sending a first webpage acquisition request including the initial identifier to the server, so that the server sends a first identifier generation algorithm to the client based on the initial identifier included in the first webpage acquisition request;

[0022] Receiving a first identifier generation algorithm sent by the server;

[0023] Generate a second identifier according to the first identifier generation algorithm; and send a second webpage acquisition request to the server, wherein the second webpage acquisition request includes the first identifier and the initial identifier, so that the server sends the second identifier generation algorithm to the client based on the first identifier and the initial identifier in the second webpage acquisition request;

[0024] Receiving a second identifier generation algorithm sent by the server;

[0025] Generate a second identifier according to the second identifier generation algorithm, the first identifier and the initial identifier; and send a third webpage acquisition request to the server, wherein the third webpage acquisition request includes the first identifier, the second identifier and the initial identifier, so that the server sends the webpage content requested by the client to the client based on the first identifier, the second identifier and the initial identifier in the third webpage acquisition request;

[0026] Receive the webpage content sent by the server.

[0027] Optionally, before sending the second webpage acquisition request to the server, the method further includes:

[0028] When it is determined that the initial identifier is not obtained, sending the first webpage acquisition request not including the initial identifier to the server, so that the server determines the initial identifier based on the first webpage acquisition request not including the initial identifier, and sends the initial identifier and the first identifier generation algorithm to the client;

[0029] Receiving the first identifier generation algorithm and the initial identifier sent by the server;

[0030] Generate a first identifier according to the first identifier generation algorithm.

[0031] Optionally, the server sends the first identifier generation algorithm to the client, specifically including:

[0032] The server uses a code reinforcement technology to reinforce the first identifier generation algorithm to obtain a reinforced first identifier generation algorithm, and sends the reinforced first identifier generation algorithm to the client;

[0033] Receiving a first identifier generation algorithm sent by the server specifically includes:

[0034] Receive the reinforced first identifier generation algorithm sent by the server.

[0035] This specification provides an anti-crawler device, which is located on a server; the device includes:

[0036] A first receiving module, used to receive a first webpage acquisition request sent by a client;

[0037] an algorithm sending module, configured to, when determining that the first webpage acquisition request includes an initial identifier, send a first identifier generation algorithm to the client, so that the client generates a first identifier according to the first identifier generation algorithm, and cause the client to send a second webpage acquisition request, wherein the second webpage acquisition request includes the initial identifier and the first identifier;

[0038] a second receiving module, configured to receive the second webpage acquisition request, and when determining that the second webpage acquisition request includes the initial identifier and the first identifier, send a second identifier generation algorithm to the client, so that the client generates a second identifier based on the initial identifier and the first identifier through the second identifier generation algorithm, and cause the client to send a third webpage acquisition request, wherein the third webpage acquisition request includes the initial identifier, the first identifier, and the second identifier;

[0039] The webpage content sending module is used to receive the third webpage acquisition request, and when it is determined that the third webpage acquisition request includes the initial identifier, the first identifier and the second identifier, send the webpage content requested by the client to the client.

[0040] Optionally, the device further comprises a determination module;

[0041] The determination module is specifically used to, when it is determined that the first web page acquisition request does not include the initial identifier, determine the initial identifier; and send the initial identifier and the first identifier generation algorithm to the client, so that the client generates the first identifier according to the first identifier generation algorithm, and causes the client to send a second web page acquisition request, wherein the second web page acquisition request includes the first identifier and the initial identifier.

[0042] Optionally, the algorithm sending module is specifically used to use code reinforcement technology to reinforce the first identifier generation algorithm to obtain a reinforced first identifier generation algorithm; and send the obtained reinforced first identifier generation algorithm to the client;

[0043] The second receiving module is specifically used to use code reinforcement technology to reinforce the second identifier generation to obtain a reinforced second identifier generation algorithm; and send the obtained reinforced second identifier generation algorithm to the client.

[0044] This specification provides an anti-crawler device, which is located at a client; the device includes:

[0045] A first request sending module, configured to send a first webpage acquisition request including the initial identifier to a server when it is determined that the initial identifier is acquired, so that the server sends a first identifier generation algorithm to the client based on the initial identifier included in the first webpage acquisition request;

[0046] A first algorithm receiving module, used to receive a first identifier generation algorithm sent by the server;

[0047] A second request sending module is configured to generate a second identifier according to the first identifier generation algorithm; and send a second webpage acquisition request to the server, wherein the second webpage acquisition request includes the first identifier and the initial identifier, so that the server sends the second identifier generation algorithm to the client based on the first identifier and the initial identifier in the second webpage acquisition request;

[0048] A second algorithm receiving module, used to receive a second identifier generation algorithm sent by the server;

[0049] a third request sending module, configured to generate a second identifier according to the second identifier generation algorithm, the first identifier and the initial identifier; and send a third webpage acquisition request to the server, wherein the third webpage acquisition request includes the first identifier, the second identifier and the initial identifier, so that the server sends the webpage content requested by the client to the client based on the first identifier, the second identifier and the initial identifier in the third webpage acquisition request;

[0050] The webpage content receiving module is used to receive the webpage content sent by the server.

[0051] Optionally, the device further comprises: a determination module;

[0052] The determination module is specifically used to, when it is determined that the initial identifier has not been obtained, send a first web page acquisition request that does not include the initial identifier to the server, so that the server determines the initial identifier based on the first web page acquisition request that does not include the initial identifier, and sends the initial identifier and a first identifier generation algorithm to the client; receive the first identifier generation algorithm and the initial identifier sent by the server; and generate a first identifier according to the first identifier generation algorithm.

[0053] Optionally, the first request sending module is specifically used for the server to use code reinforcement technology to reinforce the first identifier generation algorithm to obtain the reinforced first identifier generation algorithm, and send the reinforced first identifier generation algorithm to the client;

[0054] The first algorithm receiving module is specifically used to receive the reinforced first identifier generation algorithm sent by the server.

[0055] This specification provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned anti-crawler method is implemented.

[0056] This specification provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned anti-crawler method when executing the program.

[0057] At least one of the above technical solutions adopted in this specification can achieve the following beneficial effects:

[0058] In the anti-crawler method provided in this specification, after the server receives the first web page acquisition request from the client, when it is determined that the first web page acquisition request includes an initial identifier, the first identifier generation algorithm is sent to the client. The client can execute the first identifier generation algorithm to obtain the first identifier, and send a second web page acquisition request containing the first identifier and the initial identifier to the server. The server verifies the first identifier and the initial identifier in the second web page acquisition request, and when the verification is successful, the second identifier generation algorithm can be sent to the client. The client can execute the second identifier generation algorithm, obtain the second identifier based on the first identifier and the initial identifier, and send a third web page acquisition request containing the first identifier, the second identifier and the initial identifier to the server. The server can verify the first identifier, the second identifier and the initial identifier in the third web page acquisition request, and when the verification is successful, the server sends the web page content requested by the client to the client.

[0059] It can be seen that in this method, the server can verify the initial identification, which is used to identify and verify the identity of the client communicating with the server. The server can verify the first identification, and the first identification is obtained by the client through the execution of the first identification generation algorithm sent by the server. Since most crawlers cannot execute the first identification generation algorithm, or in other words, the crawler needs a certain cost to execute the first identification generation algorithm, the crawler can be preliminarily screened through the first identification and the initial identification to identify normal users. In addition, after verifying the first identification and the initial identification, the server can also verify the second identification. Since the second identification is obtained by the client through the execution of the second identification generation algorithm sent by the server, and based on the first identification and the initial identification, the correlation between the first identification and the initial identification is added, thereby preventing the first identification or the second identification from being tampered with, and strengthening the screening of crawlers. On the whole, this method can effectively suppress illegal web page acquisition requests and crawler behaviors, ensure that the requests at the client are requests sent by normal users, and improve the security of communication between the server and the client. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] The drawings described herein are used to provide a further understanding of this specification and constitute a part of this specification. The exemplary embodiments and descriptions of this specification are used to explain this specification and do not constitute an improper limitation on this specification.

[0061] In the figure:

[0062] Figure 1 A flowchart of an anti-crawler method in this specification;

[0063] Figure 2 A schematic diagram of the interaction between a server and a client provided in this manual;

[0064] Figure 3 A flowchart of an anti-crawler method in this specification;

[0065] Figure 4 A schematic diagram of an anti-crawler device provided in this specification;

[0066] Figure 5 A schematic diagram of an anti-crawler device provided in this specification;

[0067] Figure 6 The corresponding Figure 1 as well as Figure 2 Schematic diagram of electronic equipment. DETAILED DESCRIPTION

[0068] In order to make the purpose, technical solutions and advantages of this specification more clear, the technical solutions of this specification will be clearly and completely described below in combination with the specific embodiments of this specification and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of this specification, not all of them. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this specification.

[0069] In addition, it should be noted that all actions of acquiring signals, information or data in this manual are performed in compliance with the relevant local data protection laws and policies and with the authorization given by the owner of the corresponding device.

[0070] It should be noted that, in the absence of conflict, the features in the following embodiments and implementations may be combined with each other.

[0071] The technical solutions provided by the embodiments of this specification are described in detail below in conjunction with the accompanying drawings.

[0072] Figure 1 A flowchart of an anti-crawler method provided in this specification.

[0073] S100: Receive a first webpage acquisition request sent by a client.

[0074] S102: When it is determined that the first web page acquisition request includes an initial identifier, a first identifier generation algorithm is sent to the client, so that the client generates a first identifier according to the first identifier generation algorithm, and the client sends a second web page acquisition request, wherein the second web page acquisition request includes the initial identifier and the first identifier.

[0075] like Figure 2 As shown, it is a schematic diagram of the interaction between a server and a client provided in this specification. It can be seen that in the anti-crawler method provided in this specification, three interactions are performed between the client and the server. Specifically, in this specification, when the client accesses the target web page, a first web page acquisition request can be sent to the server. The server can receive the first web page acquisition request sent by the client. In addition, the server can parse the first web page acquisition request to determine whether the first web page acquisition request includes an initial identifier. When it is determined that the first web page acquisition request includes an initial identifier, the server can send a first identifier generation algorithm to the client. The client can then receive the first identifier generation algorithm, and can execute the first identifier generation algorithm to generate a first identifier, and can send a second web page acquisition request to the server, and the second web page acquisition request includes the initial identifier and the first identifier.

[0076] In one or more embodiments of the present specification, when the server determines that the first webpage acquisition request does not include the initial identifier, the server can determine the initial identifier and send the initial identifier and the first identifier generation algorithm to the client, and the client can receive the initial identifier and the first identifier generation algorithm. Thus, the first identifier can be generated according to the first identifier generation algorithm, and a second webpage acquisition request can be sent to the server, and the second webpage acquisition request includes the first identifier and the initial identifier.

[0077] It should be noted that this specification does not limit the initial identifier, which can be a character, a number, etc. The role of the initial identifier is to enable the server to mark the client and to be used for subsequent verification. In addition, in one or more embodiments of this specification, the first identifier generation algorithm can be a random number generation algorithm, or a JavaScript script for generating the first identifier, saving the first identifier to the page acquisition request, and triggering a page refresh, etc. This specification does not limit this either, as long as the first identifier generation algorithm can generate the first identifier.

[0078] In one or more embodiments of the present specification, the first web page acquisition request sent by the client may be an HTTP request, and the HTTP request includes a cookie. The server may determine whether the cookie in the request includes an initial identifier. When it is determined that the initial identifier exists, the server may send a first identifier generation algorithm to the client. Wherein, when the first identifier generation algorithm is a JavaScript script, in order to distinguish, the JavaScript script corresponding to the first identifier generation algorithm is called a first script. The first script may generate a first identifier for the server to determine the identity of the client, and may save the first identifier in the cookie to prevent the content in the cookie from being tampered with, thereby improving the security of communication between the client and the server. The first script may also trigger a page refresh, so that the second web page acquisition request includes the first identifier. Therefore, after receiving the first script, the client may execute the first script to generate the first identifier, and may generate a second web page acquisition request through the first script, and the second web page acquisition request includes the first identifier and the initial identifier.

[0079] When the server determines that the initial identifier does not exist in the cookie, the server can determine the initial identifier and send the initial identifier and the first script to the client. The client can save the initial identifier in the cookie, execute the first script to generate the first identifier, save the first identifier in the cookie, and trigger a page refresh, so that the second web page acquisition request sent to the server includes the first identifier and the initial identifier.

[0080] Among them, when determining the initial identifier, the server may use a random number generation algorithm, etc. The initial identifier may be pre-set, and this specification does not make any specific restrictions.

[0081] In addition, the first identifier generation algorithm may also be used only to generate the first identifier, and the client saves the first identifier in a Cookie by itself, and generates and sends a second page acquisition request including the first identifier and the initial identifier.

[0082] S104: Receive the second web page acquisition request, and when it is determined that the second web page acquisition request includes the initial identifier and the first identifier, send a second identifier generation algorithm to the client, so that the client generates a second identifier based on the initial identifier and the first identifier through the second identifier generation algorithm, and causes the client to send a third web page acquisition request, wherein the third web page acquisition request includes the initial identifier, the first identifier, and the second identifier.

[0083] Therefore, the server can receive the second web page acquisition request and determine whether the second web page acquisition request includes the initial identifier and the first identifier. When it is determined that the second web page acquisition request includes the first identifier and the initial identifier, the second identifier generation algorithm can be sent to the client. The client can execute the second identifier generation algorithm and generate the second identifier based on the initial identifier and the first identifier, so that a third web page acquisition request including the initial identifier, the first identifier and the second identifier can be sent to the server.

[0084] It should be noted that, in one or more embodiments of the present specification, the second identifier generation algorithm may be a hash algorithm, or a JavaScript script for generating a second identifier based on the first identifier and the initial identifier, saving the second identifier in a page acquisition request, and triggering a page refresh, etc. The present specification does not impose any restrictions on this, as long as the second identifier generation algorithm can generate a second identifier.

[0085] In one or more embodiments of the present specification, the second web page acquisition request sent by the client may be an HTTP request, and the server may determine whether the Cookie in the request contains the initial identifier and the first identifier. When it is determined that the first identifier and the initial identifier exist, the server may send the second identifier generation algorithm to the client. Wherein, when the second identifier generation algorithm is a JavaScript script, in order to distinguish, the JavaScript script corresponding to the second identifier generation algorithm is called a second script, and the second script may generate a second identifier based on the first identifier and the initial identifier, and may save the second identifier to the Cookie, and may also trigger a page refresh, so that the third web page acquisition request includes the second identifier. Thus, after receiving the second script, the client may execute the second script, thereby generating the second identifier based on the first identifier and the initial identifier, and may generate a third web page acquisition request through the second script, and the third web page acquisition request includes the first identifier, the second identifier and the initial identifier.

[0086] When the server determines that the Cookie of the second web page acquisition request does not contain any one of the first identifier and the initial identifier, the server may determine that the client is a crawler and may stop providing page content acquisition service to the client.

[0087] In addition, the second identifier generation algorithm can also be used only to generate the second identifier. The client saves the second identifier in the Cookie by itself, and generates and sends a third page acquisition request including the first identifier, the second identifier and the initial identifier.

[0088] S106: receiving the third webpage acquisition request, and when it is determined that the third webpage acquisition request includes the initial identifier, the first identifier, and the second identifier, sending the webpage content requested by the client to the client.

[0089] After receiving the third web page acquisition request sent by the client, the server can determine whether the third web page acquisition request includes the first identifier, the second identifier and the initial identifier. When it is determined that the third web page acquisition request includes the first identifier, the second identifier and the initial identifier, the server can determine that the client is a non-crawler, that is, a normal user. The server can then determine the web page content requested by the client and send the determined web page content requested by the client to the client.

[0090] When the server determines that the Cookie of the third web page acquisition request does not contain any of the first identifier, the second identifier and the initial identifier, the server may determine that the client is a crawler and may stop providing page content acquisition services to the client.

[0091] Based on the above Figure 1 The anti-crawler method can be seen that in this method, the server can verify the initial identification, which is used to identify and verify the identity of the client communicating with the server. The server can verify the first identification, and the first identification is obtained by the client through the execution of the first identification generation algorithm sent by the server. Since most crawlers cannot execute the first identification generation algorithm, or in other words, the crawler needs a certain cost to execute the first identification generation algorithm, the crawler can be preliminarily screened through the first identification and the initial identification to identify normal users. In addition, after verifying the first identification and the initial identification, the server can also verify the second identification. Since the second identification is obtained by the client through the execution of the second identification generation algorithm sent by the server, and according to the first identification and the initial identification, the correlation between the first identification and the initial identification is added, thereby preventing the first identification or the second identification from being tampered with, and strengthening the screening of crawlers. On the whole, this method can effectively suppress illegal web page acquisition requests and crawler behavior, ensure that the request at the client is a request sent by a normal user, and improve the security of communication between the server and the client.

[0092] In addition, in order to improve the anti-crawler strength, in the above step S102, when the server sends the first identification generation algorithm to the client, the server can use code reinforcement technology to reinforce the first identification generation algorithm to obtain the reinforced first identification generation algorithm, and send the reinforced first identification generation algorithm to the client. And in the above step S104, when the server sends the second identification generation algorithm to the client, the server can use code reinforcement technology to reinforce the second identification generation to obtain the reinforced second identification generation algorithm, and send the reinforced second identification generation algorithm to the client. Strengthening the first identification generation algorithm and the second identification generation algorithm can increase the cost of the crawler to interpret and execute the first identification algorithm and the second identification generation algorithm, thereby improving the anti-crawler strength.

[0093] Furthermore, in order to improve the strength of anti-crawler, in one or more embodiments of this specification, when the server sends the first identifier generation algorithm and the second identifier generation algorithm to the client, it can also send an execution condition to the client, and the execution condition is used to trigger the client to execute the first identifier generation algorithm and the second identifier generation algorithm, that is, only when the execution condition is met, the client can execute the first identifier generation algorithm and the second identifier generation algorithm. The execution condition can be: the operating environment of the client is a real client operating environment, that is, the operating environment of the client is not a virtual machine or a third-party operating framework.

[0094] That is to say, in the above step S100, the server may send the first identifier generation algorithm and the first execution condition to the client, the first execution condition is used to trigger the client to execute the first identifier generation algorithm, and the first execution condition is: the operating environment of the client is not a virtual machine or a third-party operating framework, when the client satisfies the first execution condition, the client can execute the first identifier generation algorithm, that is, the client can generate the first identifier through the first identifier generation algorithm. In the above step S102, the server may send the second identifier generation algorithm and the second execution condition to the client, the second execution condition is used to trigger the client to execute the second identifier generation algorithm, and the second execution condition is: the operating environment of the client is not a virtual machine or a third-party operating framework, when the client satisfies the second execution condition, the client can execute the second identifier generation algorithm, that is, the client can generate the second identifier based on the first identifier and the initial identifier through the second identifier generation algorithm.

[0095] In addition, based on Figure 1 as well as Figure 2 ,This specification also provides an anti-crawler method applied to the client, such as Figure 3 As shown, Figure 3This is a flowchart of an anti-crawler method provided in this specification. The specific process is as follows:

[0096] S300: When it is determined that the initial identifier is obtained, a first webpage acquisition request including the initial identifier is sent to a server, so that the server sends a first identifier generation algorithm to the client based on the initial identifier included in the first webpage acquisition request.

[0097] S302: Receive a first identifier generation algorithm sent by the server.

[0098] S304: Generate a second identifier according to the first identifier generation algorithm; and send a second web page acquisition request to the server, wherein the second web page acquisition request includes the first identifier and the initial identifier, so that the server sends the second identifier generation algorithm to the client based on the first identifier and the initial identifier in the second web page acquisition request.

[0099] S306: Receive a second identifier generation algorithm sent by the server.

[0100] S308: Generate a second identifier according to the second identifier generation algorithm, the first identifier and the initial identifier; and send a third web page acquisition request to the server, wherein the third web page acquisition request includes the first identifier, the second identifier and the initial identifier, so that the server sends the web page content requested by the client to the client based on the first identifier, the second identifier and the initial identifier in the third web page acquisition request.

[0101] S310: Receive the webpage content sent by the server.

[0102] In the above steps S300-S302, when the client determines that the initial identifier is not obtained, it can be determined that the client is accessing the server for the first time, and the client can send a first webpage acquisition request that does not include the initial identifier to the server, and the server can determine the initial identifier based on the first webpage acquisition request that does not include the initial identifier, and send the initial identifier and the first identifier generation algorithm to the client. The client can receive the first identifier generation algorithm and the initial identifier sent by the server, and generate the first identifier according to the first identifier generation algorithm.

[0103] In addition, as mentioned above, the server can strengthen the first identification generation algorithm and the second identification generation algorithm, and send the strengthened first identification generation algorithm and the second identification generation algorithm to the client. The client can receive the strengthened first identification generation algorithm and the strengthened second identification generation algorithm, thereby improving the anti-crawler strength.

[0104] The above is an anti-crawler method provided by one or more embodiments of this specification. Based on the same idea, this specification also provides a corresponding anti-crawler device, such as Figure 4 shown.

[0105] Figure 4 A schematic diagram of an anti-crawler device provided in this specification, wherein the device is located on a server; the device comprises:

[0106] The first receiving module 400 is used to receive a first webpage acquisition request sent by a client;

[0107] an algorithm sending module 402, configured to send a first identifier generation algorithm to the client when it is determined that the first webpage acquisition request includes an initial identifier, so that the client generates a first identifier according to the first identifier generation algorithm, and to cause the client to send a second webpage acquisition request, wherein the second webpage acquisition request includes the initial identifier and the first identifier;

[0108] The second receiving module 404 is configured to receive the second webpage acquisition request, and when it is determined that the second webpage acquisition request includes the initial identifier and the first identifier, send a second identifier generation algorithm to the client, so that the client generates a second identifier based on the initial identifier and the first identifier through the second identifier generation algorithm, and causes the client to send a third webpage acquisition request, wherein the third webpage acquisition request includes the initial identifier, the first identifier, and the second identifier;

[0109] The webpage content sending module 406 is configured to receive the third webpage acquisition request, and when it is determined that the third webpage acquisition request includes the initial identifier, the first identifier, and the second identifier, send the webpage content requested by the client to the client.

[0110] Optionally, the apparatus further includes a determination module 408;

[0111] The determination module 408 is specifically used to, when it is determined that the first web page acquisition request does not include the initial identifier, determine the initial identifier; and send the initial identifier and the first identifier generation algorithm to the client, so that the client generates a first identifier according to the first identifier generation algorithm, and causes the client to send a second web page acquisition request, wherein the second web page acquisition request includes the first identifier and the initial identifier.

[0112] Optionally, the algorithm sending module 402 is specifically used to use code reinforcement technology to reinforce the first identifier generation algorithm to obtain a reinforced first identifier generation algorithm; and send the obtained reinforced first identifier generation algorithm to the client;

[0113] The second receiving module 404 is specifically used to use code reinforcement technology to reinforce the second identifier generation to obtain a reinforced second identifier generation algorithm; and send the obtained reinforced second identifier generation algorithm to the client.

[0114] Figure 5 A schematic diagram of an anti-crawler device provided in this specification, wherein the device is located at a client; the device includes:

[0115] A first request sending module 500 is used to send a first webpage acquisition request including the initial identifier to the server when it is determined that the initial identifier is obtained, so that the server sends a first identifier generation algorithm to the client based on the initial identifier included in the first webpage acquisition request;

[0116] A first algorithm receiving module 502, configured to receive a first identifier generation algorithm sent by the server;

[0117] A second request sending module 504 is configured to generate a second identifier according to the first identifier generation algorithm; and send a second webpage acquisition request to the server, wherein the second webpage acquisition request includes the first identifier and the initial identifier, so that the server sends the second identifier generation algorithm to the client based on the first identifier and the initial identifier in the second webpage acquisition request;

[0118] A second algorithm receiving module 506, configured to receive a second identifier generating algorithm sent by the server;

[0119] A third request sending module 508 is configured to generate a second identifier according to the second identifier generation algorithm, the first identifier and the initial identifier; and send a third webpage acquisition request to the server, wherein the third webpage acquisition request includes the first identifier, the second identifier and the initial identifier, so that the server sends the webpage content requested by the client to the client based on the first identifier, the second identifier and the initial identifier in the third webpage acquisition request;

[0120] The webpage content receiving module 510 is used to receive the webpage content sent by the server.

[0121] Optionally, the device further includes: a determination module 512;

[0122] The determination module 512 is specifically used to, when it is determined that the initial identifier has not been obtained, send a first web page acquisition request that does not include the initial identifier to the server, so that the server determines the initial identifier based on the first web page acquisition request that does not include the initial identifier, and sends the initial identifier and a first identifier generation algorithm to the client; receive the first identifier generation algorithm and the initial identifier sent by the server; and generate a first identifier according to the first identifier generation algorithm.

[0123] Optionally, the first request sending module 500 is specifically used for the server to use code reinforcement technology to reinforce the first identifier generation algorithm to obtain the reinforced first identifier generation algorithm, and send the reinforced first identifier generation algorithm to the client;

[0124] The first algorithm receiving module 502 is specifically used to receive the reinforced first identifier generation algorithm sent by the server.

[0125] This specification also provides a computer-readable storage medium, which stores a computer program, which can be used to execute the above Figure 1 as well as Figure 2 The anti-crawler method shown.

[0126] This manual also provides Figure 6 The schematic structure diagram of the electronic device shown in FIG. Figure 6 As mentioned above, at the hardware level, the electronic device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory, and may also include other hardware required for the business. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to achieve the above Figure 1 as well as Figure 2 The anti-crawler method shown.

[0127] Of course, in addition to software implementation, this specification does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is to say, the executor of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0128] In the 1990s, improvements to a technology could be clearly distinguished as hardware improvements (for example, improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the method flow). However, with the development of technology, many improvements to the method flow today can be regarded as direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that an improvement in a method flow cannot be implemented using a hardware entity module. For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is such an integrated circuit whose logical function is determined by the user's programming of the device. Designers can "integrate" a digital system on a PLD by programming it themselves, without having to ask a chip manufacturer to design and produce a dedicated integrated circuit chip. Moreover, nowadays, instead of manually making integrated circuit chips, this kind of programming is mostly implemented by "logic compiler" software, which is similar to the software compiler used when developing and writing programs, and the original code before compilation must also be written in a specific programming language, which is called hardware description language (HDL). There is not only one HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also know that it is only necessary to program the method flow slightly in the above-mentioned hardware description languages ​​and program it into the integrated circuit, and then it is easy to obtain the hardware circuit that implements the logic method flow.

[0129] The controller can be implemented in any appropriate manner, for example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (such as software or firmware) that can be executed by the (micro)processor, a logic gate, a switch, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller. Examples of controllers include, but are not limited to, the following microcontrollers: ARC625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art also know that in addition to implementing the controller in a purely computer-readable program code manner, the controller can be implemented in the form of a logic gate, a switch, an application-specific integrated circuit, a programmable logic controller, and an embedded microcontroller by logically programming the method steps. Therefore, this controller can be considered as a hardware component, and the devices included therein for implementing various functions can also be regarded as structures within the hardware component. Or even, the devices for implementing various functions can be regarded as both software modules for implementing the method and structures within the hardware component.

[0130] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0131] For the convenience of description, the above device is described in various units according to their functions. Of course, when implementing this specification, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0132] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems, or computer program products. Therefore, this specification may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0133] This specification is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of this specification. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0134] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0135] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0136] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0137] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0138] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0139] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0140] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems or computer program products. Therefore, this specification may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0141] This specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. This specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0142] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0143] The above description is only an embodiment of the present specification and is not intended to limit the present specification. For those skilled in the art, the present specification may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present specification shall be included in the scope of the claims of the present specification.

Claims

1. A method for preventing crawlers, characterized in that: The method is applied to a server; the method comprises: Receive a first webpage acquisition request sent by a client; When it is determined that the first webpage acquisition request includes an initial identifier, sending a first identifier generation algorithm to the client, so that the client generates a first identifier according to the first identifier generation algorithm, and causing the client to send a second webpage acquisition request, wherein the second webpage acquisition request includes the initial identifier and the first identifier; receiving the second webpage acquisition request, and when determining that the second webpage acquisition request includes the initial identifier and the first identifier, sending a second identifier generation algorithm to the client, so that the client generates a second identifier based on the initial identifier and the first identifier through the second identifier generation algorithm, and causing the client to send a third webpage acquisition request, wherein the third webpage acquisition request includes the initial identifier, the first identifier, and the second identifier; The third webpage acquisition request is received, and when it is determined that the third webpage acquisition request includes the initial identifier, the first identifier, and the second identifier, the webpage content requested by the client is sent to the client.

2. The method according to claim 1, characterized in that After receiving the first webpage acquisition request sent by the client and before receiving the second webpage acquisition request, the method further includes: When it is determined that the first web page acquisition request does not include an initial identifier, an initial identifier is determined; and the initial identifier and the first identifier generation algorithm are sent to the client, so that the client generates a first identifier according to the first identifier generation algorithm, and the client sends a second web page acquisition request, wherein the second web page acquisition request includes the first identifier and the initial identifier.

3. The method according to claim 1, characterized in that Sending the first identifier generation algorithm to the client specifically includes: Using code reinforcement technology to reinforce the first identifier generation algorithm to obtain a reinforced first identifier generation algorithm; Sending the obtained reinforced first identifier generation algorithm to the client; Sending the second identifier generation algorithm to the client specifically includes: Using code reinforcement technology to reinforce the generation of the second identifier, and obtain a reinforced second identifier generation algorithm; The obtained reinforced second identifier generation algorithm is sent to the client.

4. A method for anti-crawler, characterized in that: The method is applied to a client; the method comprises: When it is determined that the initial identifier is obtained, sending a first webpage acquisition request including the initial identifier to the server, so that the server sends a first identifier generation algorithm to the client based on the initial identifier included in the first webpage acquisition request; Receiving a first identifier generation algorithm sent by the server; Generate a second identifier according to the first identifier generation algorithm; and send a second webpage acquisition request to the server, wherein the second webpage acquisition request includes the first identifier and the initial identifier, so that the server sends the second identifier generation algorithm to the client based on the first identifier and the initial identifier in the second webpage acquisition request; Receiving a second identifier generation algorithm sent by the server; Generate a second identifier according to the second identifier generation algorithm, the first identifier and the initial identifier; and send a third webpage acquisition request to the server, wherein the third webpage acquisition request includes the first identifier, the second identifier and the initial identifier, so that the server sends the webpage content requested by the client to the client based on the first identifier, the second identifier and the initial identifier in the third webpage acquisition request; Receive the webpage content sent by the server.

5. The method according to claim 4, characterized in that Before sending the second webpage acquisition request to the server, the method further includes: When it is determined that the initial identifier is not obtained, sending the first webpage acquisition request not including the initial identifier to the server, so that the server determines the initial identifier based on the first webpage acquisition request not including the initial identifier, and sends the initial identifier and the first identifier generation algorithm to the client; Receiving the first identifier generation algorithm and the initial identifier sent by the server; Generate a first identifier according to the first identifier generation algorithm.

6. The method according to claim 4, characterized in that The server sends the first identifier generation algorithm to the client, specifically including: The server uses a code reinforcement technology to reinforce the first identifier generation algorithm to obtain a reinforced first identifier generation algorithm, and sends the reinforced first identifier generation algorithm to the client; Receiving a first identifier generation algorithm sent by the server specifically includes: Receive the reinforced first identifier generation algorithm sent by the server.

7. An anti-crawler device, characterized in that: The device is located on a server; the device comprises: A first receiving module, used to receive a first webpage acquisition request sent by a client; an algorithm sending module, configured to, when determining that the first webpage acquisition request includes an initial identifier, send a first identifier generation algorithm to the client, so that the client generates a first identifier according to the first identifier generation algorithm, and cause the client to send a second webpage acquisition request, wherein the second webpage acquisition request includes the initial identifier and the first identifier; a second receiving module, configured to receive the second webpage acquisition request, and when determining that the second webpage acquisition request includes the initial identifier and the first identifier, send a second identifier generation algorithm to the client, so that the client generates a second identifier based on the initial identifier and the first identifier through the second identifier generation algorithm, and cause the client to send a third webpage acquisition request, wherein the third webpage acquisition request includes the initial identifier, the first identifier, and the second identifier; The webpage content sending module is used to receive the third webpage acquisition request, and when it is determined that the third webpage acquisition request includes the initial identifier, the first identifier and the second identifier, send the webpage content requested by the client to the client.

8. An anti-crawler device, characterized in that: The device is located at the client; the device includes: A first request sending module, configured to send a first webpage acquisition request including the initial identifier to a server when it is determined that the initial identifier is acquired, so that the server sends a first identifier generation algorithm to the client based on the initial identifier included in the first webpage acquisition request; A first algorithm receiving module, used to receive a first identifier generation algorithm sent by the server; A second request sending module is configured to generate a second identifier according to the first identifier generation algorithm; and send a second webpage acquisition request to the server, wherein the second webpage acquisition request includes the first identifier and the initial identifier, so that the server sends the second identifier generation algorithm to the client based on the first identifier and the initial identifier in the second webpage acquisition request; A second algorithm receiving module, used to receive a second identifier generation algorithm sent by the server; a third request sending module, configured to generate a second identifier according to the second identifier generation algorithm, the first identifier and the initial identifier; and send a third webpage acquisition request to the server, wherein the third webpage acquisition request includes the first identifier, the second identifier and the initial identifier, so that the server sends the webpage content requested by the client to the client based on the first identifier, the second identifier and the initial identifier in the third webpage acquisition request; The webpage content receiving module is used to receive the webpage content sent by the server.

9. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 6 is implemented.

Images