Crawler identification and prevention and control method and device, computer readable storage medium and electronic equipment

Abstract

The invention relates to a crawler identification and prevention and control method and device. The method comprises the following steps: receiving a query request sent by a front end, and identifying whether the request is a crawler request by a website back end; if the request is identified as a normal user query request, processing and querying are carried out through an access process of a normal user; if the request is identified as a crawler request, directly querying non-real-time data from a cache, assembling the queried non-real-time data into a response result, and returning the response result to the rear end of the website; and the website rear end returns the response result to the crawler program as a query result. According to the method, a set of strategy for identifying the crawler is established, the request of the crawler is dredged to the efficient cache / static file, the data needing to be crawled by the crawler is cached or staticized, the accuracy and the real-time performance of the data do not need to be guaranteed, the data can be efficiently output to the crawler, and the method and the device are convenient to use. The characteristic that crawlers cannot recognize the accuracy of crawled data is utilized, the influence of the crawlers is relieved, and efficient response to normal user request output is achieved.

Description

Technical Field

[0001] The present application belongs to the field of crawler identification and control technology, and in particular relates to a crawler identification and prevention method, device, computer-readable storage medium and electronic device. Background Art

[0002] With the rapid development of Internet technology, the content of users' information systems is widely published on the Internet for public access. This information has extremely high commercial value for information integrators and information analysis companies or individuals. In order to obtain this data, they often use web crawlers or web crawler technology to capture a large amount of website data through automated means and convert it into commercial value.

[0003] Web crawlers can cause tremendous pressure on websites by accessing them at a high frequency, causing huge loads on servers, databases, and other levels. This not only affects the response speed of the website, but can also cause the service to fail to respond or time out, seriously affecting the user experience. Therefore, the website's technology vendors or teams have to invest more resources or research technical strategies to cope with the pressure brought by crawlers.

[0004] At present, conventional technical solutions mainly prevent and control crawlers by identifying their characteristics, such as IP characteristics (including IP addresses, access frequency, etc.), and taking measures such as prohibiting access or human-computer interaction (such as login, verification code). Although these solutions are effective in the short term, as time goes by, the crawler technology team can quickly overcome these strategies and continue to crawl data. In actual applications, these strategies that are mainly based on "blocking" allow crawlers to quickly perceive that requests are being prevented and quickly adjust strategies to bypass protective measures. In addition, there is another type of strategy that does not conduct strong prevention and control on crawlers, but instead improves data acquisition efficiency by expanding server resources and precise caching to cope with large traffic access. However, this strategy has extremely high requirements for cache update technology, especially for e-commerce websites that need to update inventory information in real time. In-depth system transformation is required, and the technical cost is high, which is difficult for ordinary users to afford.

[0005] Chinese patent application CN115037526B "Anti-crawler method, device, equipment and computer storage medium" discloses an anti-crawler method, which identifies crawlers by obtaining device fingerprints through probes and provides counterfeit data to crawlers. However, in actual applications, device fingerprints are often forged in current crawler programs, and identity confirmation is unreliable. Therefore, how to provide a more efficient and accurate crawler identification and control method to reduce the pressure on website servers, improve website service quality, and reduce costs has become a technical problem that needs to be solved urgently. Summary of the invention

[0006] In order to solve the above problems, this application proposes a new crawler identification and prevention method. This application establishes a set of strategies for identifying crawlers, and at the same time, through a certain amount of storage resources, it directs the crawler's requests to efficient cache / static files. By caching or staticizing the data that the crawler needs to crawl, it can be efficiently output to the crawler without ensuring the accuracy and real-time nature of the data. By taking advantage of the fact that the crawler cannot identify the accuracy of the data it crawls, the impact of the crawler is alleviated, and an efficient response to normal user request output is achieved.

[0007] In order to achieve the above objectives, this application provides the following technical solutions:

[0008] The first aspect of the present application provides a method for identifying and controlling crawlers, the method comprising:

[0009] Receive the query request sent by the front end, and the website back end identifies whether the request is a crawler request;

[0010] If the request is identified as a normal user query request, it will be processed and queried through the normal user access process;

[0011] If the request is identified as a crawler request, the non-real-time data is directly queried from the cache, and the queried non-real-time data is assembled into a response result and returned to the website backend;

[0012] The website backend returns the response result to the crawler as the query result.

[0013] Optionally, in the method of the present application, the website backend identifies whether the request is a crawler request, including:

[0014] Determine whether the request header is a regular browser feature. If not, identify it as a crawler request.

[0015] Determine whether the request contains the default request header of the programming language. If so, identify it as a crawler request.

[0016] Determine whether there is any unusual request header content set in the request. If not, identify it as a crawler request;

[0017] Determine whether there is a special request that has called the homepage in the request. If not, identify it as a crawler request;

[0018] If the above steps are not identified as crawler requests, they are identified as normal requests.

[0019] Optionally, the present application method also includes:

[0020] A request header recognition library is established for request header recognition; the request header recognition library includes: request header features of conventional browsers and request header features of a programming language or scripting language used to write crawler programs.

[0021] Optionally, the method of the present application also includes: when writing a web page program, writing an unconventional request header, and then including the unconventional request header in the request header of each call request.

[0022] Optionally, the method of the present application also includes: setting a special page request in the home page or initialization interface, and calling it when the home page or initialization interface is opened; after the background calls the special page request, immediately storing an authentication value in the cache, and writing the authentication value into the browser cookie; the next time other requests are called, if the authentication value is not found from the cookie, or the relevant content is not found in the cache, it proves that the request is not a normal request from the page, because it skips the page operation step, and is therefore identified as a crawler request.

[0023] Optionally, in the method of the present application, the access process of the normal user includes:

[0024] The user opens the website homepage or initialization interface and calls the website backend;

[0025] The website backend stores cache and writes cookies;

[0026] When querying data on a page, the page automatically includes unconventional request headers;

[0027] The website backend identifies unusual request headers, performs data queries, writes query results to the cache, and sets the cache validity period.

[0028] Optionally, the method of the present application further includes: setting the following configurable items by programming, and realizing identification and prevention and control of crawlers by dynamically adjusting and expanding the configurable items in real time:

[0029] Request header feature items: including request header features of regular browsers and request header features of programming languages ​​or scripting languages ​​used to write crawler programs;

[0030] Unconventional request header items: include the request header name and the corresponding value. Unconventional request headers are included in the request header of each call request.

[0031] Page special request item: set page special request in the home page or initialization interface, and call it when the home page or initialization interface is opened;

[0032] Cache time item: adjust the cache time.

[0033] A second aspect of the present application provides a crawler identification and prevention device, the device comprising:

[0034] Crawler identification module: used to receive the query request sent by the front end and identify whether the request is a crawler request;

[0035] Normal request processing module: used to process and query through the normal user access process when the request is identified as a normal user query request;

[0036] Crawler request processing module: when a request is identified as a crawler request, it directly queries non-real-time data from the cache, and assembles the queried non-real-time data into a response result and returns it to the website backend;

[0037] Result processing module: used to return the response result as the query result to the crawler program.

[0038] The device implements the steps of the aforementioned crawler identification and prevention method when running.

[0039] A third aspect of the present application provides an electronic device, comprising: a memory and a processor;

[0040] Memory: used to store computer programs;

[0041] Processor: used to execute the computer program to implement the steps of the aforementioned crawler identification and prevention method.

[0042] The fourth aspect of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the aforementioned crawler identification and prevention method.

[0043] In summary, this application proposes a new crawler identification and prevention method, which has the following advantages:

[0044] 1. It solves the common problem that the more blocking strategies are used, the more traffic will be blocked, and the problem that prevention measures are not as fast as crawler measures. This method creates the illusion of satisfying crawler needs in a guiding way, slowing down the learning and mutation speed of crawlers. Most crawlers are unaware of this and will continue to crawl using crawler strategies that they believe are effective.

[0045] 2. It makes it difficult for crawlers to figure out prevention and control measures. Because crawlers can still obtain data, it is difficult for them to identify which data is valid, and it is difficult to guess which aspects of the crawler program have problems, making it impossible to perform targeted upgrades.

[0046] 3. Multiple sets of identification make it difficult for the crawler to identify which link has a problem. It can modify one aspect in a targeted manner. We can adjust the strategy of one aspect in a targeted manner, or adjust the strategy slowly, so that it thinks that the main problem lies in one point, thereby focusing its energy on one point, while our identification of other points is still effective for it.

[0047] 4. Through flexible configuration, the prevention and control strategy can be adjusted quickly, making the crawler tired of adjusting the strategy and increasing its manpower cost.

[0048] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained by the techniques indicated in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the technical solutions of the embodiments of the present application, a brief introduction will be given below to the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0050] Figure 1 A control flow chart for crawler programs in the crawler identification and control method provided in an embodiment of the present application.

[0051] Figure 2 This is an access flow chart for normal users in the crawler identification and prevention method provided in an embodiment of the present application.

[0052] Figure 3 An overall implementation flow chart of the crawler identification and prevention method provided in the embodiment of the present application.

[0053] Figure 4 A flowchart of crawler identification in the crawler identification and prevention method provided in an embodiment of the present application.

[0054] Figure 5 A structural diagram of the components of the crawler identification and prevention device provided in an embodiment of the present application.

[0055] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0056] In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0057] As used herein, the term “including” and its variations are open inclusions, ie, “including but not limited to”; the term “based on” means “based at least in part on”; and the term “one embodiment” means “at least one embodiment”.

[0058] It should be noted that the modifications of "one" and "plurality" mentioned in the present application are illustrative rather than restrictive, and those skilled in the art should understand that unless the context clearly indicates otherwise, it should be understood as "one or more".

[0059] Figure 1 The following is a prevention and control process for crawler programs in the crawler identification and prevention and control method of this application, including:

[0060] The crawler sends an HTTP request, and after the request enters the enterprise website, the enterprise website backend performs prevention and control processing;

[0061] The enterprise website backend identifies whether it is a crawler request. If so, it queries non-real-time data from the cache, assembles it into a response result and returns it to the enterprise website backend, which returns it to the crawler program as query result data (crawled worthless data).

[0062] Figure 2 The following is the access process for normal users in the crawler identification and prevention method of this application, including:

[0063] The user opens the homepage / initial page and calls the website backend;

[0064] The website backend stores the cache and writes cookies; then it queries data on the page, and the page automatically brings in unconventional request headers; the website backend identifies whether it is a crawler request, and if not, it processes it according to normal business, writes the query results to the cache, and sets the cache validity period.

[0065] Figure 3 The overall implementation process of the crawler identification and prevention method of this application is shown, including the following steps:

[0066] Receive the query request sent by the front end, and the website back end identifies whether the request is a crawler request;

[0067] If the request is identified as a normal user query request, it will be processed and queried through the normal user access process;

[0068] If the request is identified as a crawler request, the non-real-time data is directly queried from the cache, and the queried non-real-time data is assembled into a response result and returned to the website backend;

[0069] The website backend returns the response result to the crawler as the query result.

[0070] This application realizes effective identification and prevention of crawlers through crawler identification, cache / static file processing, and configurable solutions.

[0071] (I) Identifying crawlers

[0072] Figure 4 The crawler identification process in the present application method is shown. The present method uses the crawler to imitate the characteristics of web page requests to identify them, and identifies the crawler identity through the following three methods.

[0073] (1) Identification of request headers: First, the request headers of browsers on the market are summarized. The User-Agent in the general browser request headers is fixed content, indicating the version of the browser used, for example: Mozilla / 5.0 (iPhone; CPU iPhone OS16_6like Mac OS X) AppleWebKit / 605.1.15 (KHTML, like Gecko) Version / 16.6 Mobile / 15E148 Safari / 604.1. However, the default value of the crawler is: xxxjavaxxx, xxxPythonxxx, which is the header that comes with the language in which the crawler is written. Therefore, this application establishes a user header recognition library, which includes: (a) the features of the headers of commonly used browsers on the market; (b) the features of the headers that are obviously written by the crawler: keywords of the writing language such as java and python.

[0074] Design principle: Since ordinary users access the system through browsers, browsers have default request header rules when sending requests, while crawlers are generally programs written in programming languages ​​or scripting languages, which have default request headers. The characteristics of their request headers are different from those of browser request headers, so the two can be easily distinguished by identifying the request headers.

[0075] (2) When writing a web page program, an unconventional request header is specially written, for example: test-agent: liulanqi. In the request header of each call request, this "test-agent: liulanqi" will be included. If this header feature is not passed in, it is a request written by the crawler.

[0076] Design principle: When a normal user visits the website through a browser, the call request is made through the web page program. This part of the program is controlled by the technical staff who developed the website, and the unconventional request headers brought into the request can be adjusted at any time. After the crawler program takes effect, it is usually not actively adjusted. As long as the data can be crawled, the original program will be used. Since the crawler program will not add unconventional request headers to the original program according to the changes in the web page call, it is easy to be identified.

[0077] (3) Set a special page request and call it when the homepage is opened. For example, set a special page request xxx.com / verify. After the background calls this request, it immediately stores an authentication value in the cache and writes this value into the browser cookie. The next time other requests are called, if this value is not found from the cookie or the relevant content is not found in the cache, it proves that this request is not from a natural person who requested the page. Since it skips the page operation step, it is likely to be a crawler program.

[0078] Design principle: Normal users must first open the front-end interface such as the home page or other initialization interface before logging in, while the crawler does not access the page, but only crawls the requests it needs to crawl. This application adds a special request call to the home page or initialization interface. Since the crawler does not understand the purpose of this request and cannot perceive the impact on its crawled data, it still only crawls the original request. Therefore, it can be identified that those who have called special requests are normal users, while those who only call regular requests are crawlers.

[0079] (II) Cache / static file processing

[0080] The following uses cache file processing as an example to explain that static files and cache files are different in form, but the implementation principles are exactly the same. The solution is as follows:

[0081] Perform crawler identification according to the crawler identification scheme described in the previous step.

[0082] For requests identified as non-crawler requests, the business is processed and queried according to the normal process, and the query results are processed. The business parameters of the request are set as key values, the query results are stored in the cache, and the cache time is set. The cache time can be set relatively long, because there is no need to ensure the accuracy of the results, so it can be gradually extended according to the current crawling pressure.

[0083] For requests identified as crawlers, cache queries are performed directly, and the key value of the request is directly used as a parameter to load the result.

[0084] Design principle: Crawlers are generally sensitive to whether they can find results, but not to whether the results are accurate. This application reduces the loss of internal system calculations by caching and storing long-term data, while also meeting the crawling needs of crawlers. This design is different from the general caching strategy, which only queries cached data for clearly identified crawlers.

[0085] (III) Configuration Solution

[0086] The following contents are programmed as configurable items so that the strategy for dealing with crawlers can be adjusted in real time.

[0087] 1. Crawler features: including parameters of regular browser headers and header features of backend language programming. This configuration item can be dynamically adjusted and expanded in real time to identify more browser and crawler features.

[0088] 2. Set unconventional request headers: including request header names and corresponding values. Unconventional request headers can be updated dynamically. After the update, it is immediately possible to identify which existing requests are from crawlers.

[0089] 3. Set the request page: including the URL of the request. It can package new requests in real time, making it difficult for the crawler to grasp which request is used to capture its crawler behavior.

[0090] 4. Set the cache time: You can adjust the cache time to adjust it to a stable level for the system.

[0091] In order to better understand the technical solution of the present application, further explanation is given with reference to embodiments of the following scenarios.

[0092] By building a control application on the backend, effective control of crawlers can be achieved. The specific implementation steps are as follows:

[0093] 1. Adjust the service application, which can be implemented using common languages ​​such as java, python, c#, etc.

[0094] 1.1 Setting the data structure of the configuration content can be implemented through a database or a configuration center so that the configuration content can be adjusted.

[0095] It includes the following aspects:

[0096] (1) Browser header features, examples are as follows:

[0097]

[0098] (2) Programming feature keywords

[0099] Configuration Content value Processer %JAVA%,%java%,%Java%,%Python%...(adjustable)

[0100] (3) Special header settings

[0101] Header Name Header Value Test-agent (adjustable) Zhenshiyonghu (adjustable)

[0102] (4) Set up special requests

[0103] Request Name value Identifying special requests Xxx / com / xxx (adjustable)

[0104] (5) Set the cache validity period

[0105]

[0106]

[0107] 1.2 The above content is made configurable through programs or middleware.

[0108] 1.3 Issue a special request interface

[0109] (1) This interface directly generates a uuid;

[0110] (2) Write the uuid into the cache;

[0111] (3) At the same time, write this uuid into the cookie.

[0112] 1.4 Write crawler identification processing in the original system program

[0113] (1) Determine whether the browser features carried by the request are among the summarized browser features;

[0114] (2) Determine whether the header contains keywords that indicate the program's programming characteristics;

[0115] (3) Determine whether a special header is carried;

[0116] (4) Determine whether the cookie contains the uuid generated in 1.3;

[0117] (5) Set the judgment result as an identifier, A is a crawler, B is a non-crawler

[0118] 1.5 Add cache query code to the original system program

[0119] (1) If it is a crawler, proceed to this step;

[0120] (2) The main business parameters of the request are combined into key values ​​according to the established rules and the cache is called;

[0121] (3) Process the cached data and return it to the client.

[0122] 1.6 Add cache storage code to the original system

[0123] (1) If it is not a crawler, proceed to this step;

[0124] (2) Normal process query;

[0125] (3) Write the query results to the cache and set the cache validity period according to the configured time.

[0126] 2. Adjust the website front-end code

[0127] 2.1 Call a special request on the home page (or initial page). The address of this request is the configured special request address (Xxx / com / xxx).

[0128] 2.2 When calling the regular request to be protected, add the content written in the header file

[0129] In the header, add special headers, for example, add: Test-agent:Zhenshiyonghu.

[0130] 3. Configure the requested mapping

[0131] The special request is on the web server or gateway, and its external address is configured as Xxx / com / xxx (for example).

[0132] 4. Once the program is running, you can prevent crawlers based on the current strategy.

[0133] 5. Dynamically adjust your strategy when it is targeted

[0134] 5.1 Configure special headers: For example, the original special headers are:

[0135] Header Name Header Value Test-agent (adjustable) Zhenshiyonghu (adjustable)

[0136] Modify it to:

[0137] Header Name Header Value Test-agent1 (adjustable) zhenduitiaozheng (adjustable)

[0138] Adjust the special header in the front-end code, change it to the modified special header, and republish the static file to take effect.

[0139] 5.2 Adjust special requests: For example, the original special request is:

[0140] Request Name value Identifying special requests Xxx / com / xxx (adjustable)

[0141] Modify it to:

[0142] Request Name value Identifying special requests Xxx / com / yyyy

[0143] The gateway regenerates the request through the above configuration, adjusts the calling address of the front-end code to the modified address, and republishes the static file to take effect.

[0144] 5.3 What to do when the server's computation and memory are too large to handle

[0145] Increase the cache time in the configuration until the system is more stable.

[0146] 5.4 What to do when some recognition is inaccurate

[0147] (1) Update the browser header configuration and add new browser header features to the configuration

[0148]

[0149] (2) Capture the keywords of programming features from the request log and update them to this configuration

[0150] Configuration Content value Processer %JAVA%,%java%,%Java%,%Python%...

[0151] Figure 5 The present invention provides a crawler identification and prevention device, comprising:

[0152] Crawler identification module: used to receive the query request sent by the front end and identify whether the request is a crawler request;

[0153] Normal request processing module: used to process and query through the normal user access process when the request is identified as a normal user query request;

[0154] Crawler request processing module: when a request is identified as a crawler request, it directly queries non-real-time data from the cache, and assembles the queried non-real-time data into a response result and returns it to the website backend;

[0155] Result processing module: used to return the response result as the query result to the crawler program.

[0156] When the above device is running, the steps of the crawler identification and prevention method disclosed in this application are implemented.

[0157] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the device, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, program segment or a part of code, and the module, program segment or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be realized by a special hardware-based system that performs the specified function or operation, or can be realized by a combination of special hardware and computer instructions.

[0158] like Figure 6 As shown, the embodiment of the present application also discloses an electronic device, including: a processor 310, a communication interface 320, a memory 330 for storing a computer program executable by the processor, and a communication bus 340. Among them, the processor 310, the communication interface 320 and the memory 330 communicate with each other through the communication bus 340. The processor 310 implements the steps of the above-mentioned crawler identification and prevention method by running the executable computer program.

[0159] It is understandable that, in addition to the memory and the processor, the electronic device may also include an input device such as a keyboard, an output device such as a display, and other communication modules. The input device, the output device, and other communication modules communicate with the processor via an I / O interface (i.e., an input / output interface).

[0160] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including, but not limited to, object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0161] Furthermore, the present application also discloses a computer-readable storage medium. When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device can execute each step of the crawler identification and prevention method disclosed in the present application.

[0162] In the context of the present application, a computer-readable storage medium may be a tangible medium, more specific examples of which would include a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0163] In particular, according to an embodiment of the present application, the process described in the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program code for executing the crawler identification and prevention method disclosed in the present application. When the computer program is executed by the processing device, the above functions defined in the method of the embodiment of the present application are executed.

[0164] Although several specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present application. The above description is only a preferred embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present application is not limited to the technical solutions formed by a specific combination of the above technical features, but also should cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above public concept.

[0165] Those skilled in the art should also understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A method for identifying and controlling crawlers, characterized in that: The method comprises: Receive the query request sent by the front end, and the website back end identifies whether the request is a crawler request; If the request is identified as a normal user query request, it will be processed and queried through the normal user access process; If the request is identified as a crawler request, the non-real-time data is directly queried from the cache, and the queried non-real-time data is assembled into a response result and returned to the website backend; The website backend returns the response result to the crawler as the query result.

2. The method according to claim 1, characterized in that The website backend identifies whether the request is a crawler request, including: Determine whether the request header is a regular browser feature. If not, identify it as a crawler request. Determine whether the request contains the default request header of the programming language. If so, identify it as a crawler request. Determine whether there is any unusual request header content set in the request. If not, identify it as a crawler request; Determine whether there is a special request that has called the homepage in the request. If not, identify it as a crawler request; If the above steps are not identified as crawler requests, they are identified as normal requests.

3. The method according to claim 2, characterized in that The method further comprises: A request header recognition library is established for request header recognition; the request header recognition library includes: request header features of conventional browsers and request header features of a programming language or scripting language used to write crawler programs.

4. The method according to claim 2, characterized in that: The method further comprises: when writing a web page program, writing an unconventional request header, and then including the unconventional request header in the request header of each call request.

5. The method according to claim 2, characterized in that: The method also includes: setting a special page request in the home page or initialization interface, and calling it when the home page or initialization interface is opened; after the background calls the special page request, immediately storing an authentication value in the cache, and writing the authentication value into the browser cookie; when other requests are called next time, if the authentication value is not found from the cookie, or the relevant content is not found in the cache, it proves that the request is not a normal request from the page, because it skips the page operation step, and is therefore identified as a crawler request.

6. The method according to claim 1, characterized in that The access process of the normal user includes: The user opens the website homepage or initialization interface and calls the website backend; The website backend stores cache and writes cookies; When querying data on a page, the page automatically includes unconventional request headers; The website backend identifies unusual request headers, performs data queries, writes query results to the cache, and sets the cache validity period.

7. The method according to claim 1, characterized in that The method further includes: setting the following configurable items by programming, and realizing identification and prevention and control of crawlers by dynamically adjusting and expanding the configurable items in real time: Request header feature items: including request header features of regular browsers and request header features of programming languages ​​or scripting languages ​​used to write crawler programs; Unconventional request header items: include the request header name and the corresponding value. Unconventional request headers are included in the request header of each call request. Page special request item: set page special request in the home page or initialization interface, and call it when the home page or initialization interface is opened; Cache time item: adjust the cache time.

8. A crawler identification and control device, characterized in that: The device comprises: Crawler identification module: used to receive the query request sent by the front end and identify whether the request is a crawler request; Normal request processing module: used to process and query through the normal user access process when the request is identified as a normal user query request; Crawler request processing module: when a request is identified as a crawler request, it directly queries non-real-time data from the cache, and assembles the queried non-real-time data into a response result and returns it to the website backend; Result processing module: used to return the response result as the query result to the crawler program.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the crawler identification and prevention method according to any one of claims 1 to 7 are implemented.

10. An electronic device, characterized in that: include: Memory and processor; Memory: used to store computer programs; Processor: used to execute the computer program to implement the steps of the crawler identification and prevention method as described in any one of claims 1-7.

Images