A method for identifying and defending against trusted crawlers

By constructing front-end static resources and gateway encryption judgment, combined with static resource interference traps, the accuracy problem of trusted crawler identification and defense is solved, the cracking difficulty and resource cost are reduced, and the identification rate is improved.

CN118573449BActive Publication Date: 2024-11-22ZHOUPU DATA TECH NANJING CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410740310.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-06-07
Publication Date
2024-11-22
Estimated Expiration
2044-06-07

AI Technical Summary

Technical Problem

Existing technologies struggle to accurately identify and defend against trusted web crawlers, easily leading to the mistaken blocking of legitimate users and impacting website security and data integrity.

Method used

By constructing front-end static resources, coloring interface requests, and performing encrypted value judgment and coloring at the gateway, combined with static resource interference traps, the accuracy of identification and the effectiveness of defense are improved.

Benefits of technology

It enables rapid and accurate identification of authorized web crawler requests, reduces the difficulty of cracking, reduces resource costs, improves the recognition rate, and distinguishes between web crawlers and normal users.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118573449B_ABST
    Figure CN118573449B_ABST
Patent Text Reader

Abstract

The present invention provides a method for identifying and defending against trusted crawlers, which is characterized by comprising the following steps: 1) constructing front-end static resources; 2) coloring a user interface; 3) a gateway analyzing, judging and coloring a user interface request, and determining whether to release or reject the user interface request according to the coloring result; 4) constructing a static resource interference trap; by changing the encrypted field and the trap script each time according to the agreed encryption range, the interface request of the trusted crawler can be quickly and accurately identified and the request can be defended, which improves the recognition rate and also increases the cracking difficulty of the crawler; as long as the static resource is released once, the encrypted field, encrypted file and other contents originally cracked by the crawler will become invalid, and at the same time, the defender only needs to release the static resource once, and the remaining configuration items are all dynamically stored, and there is no need to restart the server, and the resource cost spent is low.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of crawler identification and defense technology, and in particular to a trusted crawler identification and defense method. Background Art

[0002] A trusted crawler refers to a program or script that has obtained a username, password or login ID through normal channels, uses a crawler to obtain data and performs secondary processing or analysis. As the crawled party, regardless of whether the crawler is trusted or not, it is particularly important for the service provider to protect the security of its own sensitive data.

[0003] Crawler identification is usually performed by checking HTTP request header information, user agent strings, and access patterns. Common techniques include using CAPTCHA (Completely Automated Turing Test to Tell Computers and Humans Apart) or implementing IP address restrictions. These methods can help website administrators determine which visits are initiated by legitimate users and which are initiated by crawlers, so that they can take appropriate measures to protect the security of the website and the integrity of the data.

[0004] Correspondingly, there are many ways to prohibit crawlers, such as:

[0005] IP address restriction: Websites can block crawlers by restricting access to specific IP addresses. This approach may not be flexible enough because it may affect access by normal users.

[0006] User agent detection: Websites can check the user agent string of HTTP requests to identify the agent used by crawlers. By identifying common crawler user agent strings, websites can deny them access.

[0007] CAPTCHA: Websites can require users to enter a CAPTCHA when accessing sensitive pages or frequent visits. This method can effectively prevent automated crawling because crawlers usually cannot handle CAPTCHAs.

[0008] Frequency Limitation: Websites can limit the frequency of requests from the same IP address or user within a certain period of time. This prevents crawlers from accessing the website too frequently, thereby reducing the load on the server.

[0009] JavaScript detection: Websites can verify whether they are real users by asking users to execute JavaScript code. Because most crawlers do not execute JavaScript, this method can effectively distinguish between crawlers and real users.

[0010] Dynamically generated content: Websites can use JavaScript or server-side technologies to dynamically generate content, making it difficult for crawlers to capture valid information on the page.

[0011] Since the crawler is a trusted crawler (the user name and password were obtained through legal means), and after analysis, the technical means it uses are mostly a combination of web page crawling trusted tokens and interface crawling sensitive data, the above-listed technical means for identifying and prohibiting crawlers cannot be used directly.

[0012] The identification of trusted crawlers is different from that of ordinary crawlers. It is necessary to combine the subsequent operations for identification and defense. Otherwise, it is likely that the identification will fail or be wrong, which will affect the normal use of customers. For example, with IP address access identification and restriction, it is possible that the trusted crawler crawls the machine and the normal use customer uses the same IP access exit. If a single blocked IP is blocked, it will be blocked by mistake, affecting normal use customers.

[0013] Based on this, it is necessary to provide a trusted crawler identification and defense method that can not only correctly identify the trusted crawler, but also carry out targeted and precise defense against the crawler. Summary of the invention

[0014] In order to solve the defects and shortcomings in the prior art, the present invention provides a method for identifying and defending against trusted crawlers.

[0015] The specific scheme provided by the present invention is:

[0016] A method for identifying and defending against a trusted crawler, characterized in that it comprises the following steps:

[0017] 1) Build front-end static resources;

[0018] 2) Coloring user interface requests;

[0019] 3) The gateway analyzes, judges and marks the user's interface request, and decides whether to release or reject the user's interface request based on the marking result;

[0020] 4) Build a static resource interference trap.

[0021] As a further preferred embodiment of the present invention, the step 1) comprises the following steps:

[0022] 1.1) Initialize the set of encryptable fields;

[0023] 1.2) Agree on the file names and variable names of the encrypted files and the encrypted fields defined in the encrypted files;

[0024] 1.3) Randomly select several fields from the field set in step 1.1);

[0025] 1.4) Record the randomly selected fields in the storage medium;

[0026] 1.5) Replace the value of the variable name agreed in step 1.2) with the value of the field variable name obtained in step 1.3);

[0027] 1.6) Complete the subsequent project construction.

[0028] As a further preferred embodiment of the present invention, the step 1.1) comprises the following steps:

[0029] 1.11) Analyze all request interfaces or sensitive interfaces of the system;

[0030] 1.12) Extract the common fields in the analyzed interface;

[0031] 1.13) The interfaces meeting the preset conditions are stored in a storage medium in the form of a list.

[0032] As a further preferred embodiment of the present invention, the storage medium is a database or a file.

[0033] As a further preferred embodiment of the present invention, the step 2) comprises the following steps:

[0034] 2.1) Get the field to be encrypted from the encrypted script;

[0035] 2.2) Pre-read the value of the encrypted field in the request and compose it into a string;

[0036] 2.3) Encrypt the composed string;

[0037] 2.4) Add the key value to the request header and assign it the encrypted value obtained in step 2.3);

[0038] 2.5) Send an interface request to the gateway.

[0039] As a further preferred embodiment of the present invention, the step 3) comprises the following steps:

[0040] 3.1) The gateway intercepts the interface request and reads the encrypted value of the interface request;

[0041] 3.2) Compose the encrypted value into a string;

[0042] 3.3) Encrypt the string;

[0043] 3.4) Read the key value agreed in the Header value in the interface request;

[0044] 3.5) Determine whether the encrypted values ​​obtained in step 3.3) and step 3.4) are equal, and perform corresponding color marking;

[0045] 3.6) Determine whether to release the interface request based on the staining result obtained in step 3.5);

[0046] 3.7) Release or reject the interface request accordingly.

[0047] As a further preferred embodiment of the present invention, in step 3.5),

[0048] When the judgment results are equal, the interface request is marked with a color;

[0049] When the judgment results are not equal, the interface request is not marked.

[0050] As a further preferred embodiment of the present invention, in step 3.6),

[0051] When the interface request has a dye mark, the interface request is allowed to be released;

[0052] When the interface request does not have a dye mark, the interface request is rejected.

[0053] As a further preferred embodiment of the present invention, the step 4) comprises the following steps:

[0054] 4.1) Repeat the steps in step 1) to build the front-end static resources;

[0055] 4.2) Generate several sets of random strings;

[0056] 4.3) Change the file names of the puppet script and the encrypted script to the file names of the random strings generated in step 4.2);

[0057] 4.4) Update the mapping relationship between script files and file names.

[0058] As a further preferred embodiment of the present invention, in step 4.2), the random string is generated using openssl or shell script.

[0059] Compared with the prior art, the present invention can achieve the following technical effects:

[0060] 1) The present invention provides a trusted crawler identification and defense method. By establishing an agreed encryption range, changing the encryption field each time, and changing the trap script, the interface request of the trusted crawler can be quickly and accurately identified and the request can be defended. This improves the recognition rate while also increasing the difficulty of cracking by the crawler.

[0061] 2) The present invention provides a trusted crawler identification and defense method. As long as the static resources are released once, the encrypted fields, encrypted files and other contents originally cracked by the crawler will become invalid. At the same time, the defender only needs to release the static resources once, and the remaining configuration items are all dynamically stored. There is no need to restart the server, and the resource cost is low.

[0062] 3) The present invention provides a method for identifying and defending against trusted crawlers. The technical solution proposed in the present invention can more accurately identify crawlers and normal users, and use smaller resources to deal with the processing solution after the crawler is cracked, which has advantages in accurate identification and resource investment ratio. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 A flowchart of the steps of the method provided by the present invention.

[0064] Figure 2 This is a flow chart of step 1) in the method provided by the present invention.

[0065] Figure 3 This is a flow chart of step 2) in the method provided by the present invention.

[0066] Figure 4 This is a flow chart of step 3) in the method provided by the present invention.

[0067] Figure 5 This is a flow chart of step 4) in the method provided by the present invention. DETAILED DESCRIPTION

[0068] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0069] In the description of the present invention, it should be noted that the terms "upper", "lower", "inner", "outer", "front end", "rear end", "two ends", "one end", "the other end" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance.

[0070] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "provided with", "connected", etc. should be understood in a broad sense. For example, "connected" can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0071] [First embodiment]

[0072] like Figure 1 The first embodiment of the present invention is shown, which provides a method for identifying and defending against trusted crawlers, including the following steps:

[0073] 1) Build front-end static resources; such as Figure 2 As shown, the specific steps include:

[0074] 1.1) Initialize the set of encryptable fields; need to: analyze all HTTP request interfaces of the system or information-sensitive interfaces that need special attention; then extract the common fields in the analyzed interfaces; then store the interfaces that meet the preset conditions in the form of a list in a storage medium, which can be a database, a file or other form.

[0075] 1.2) Agree on the file names and variable names of the encrypted files and the encrypted fields defined in the encrypted files;

[0076] 1.3) At this point, the construction begins. The field set obtained in step 1.1) is obtained through a shell script, and several fields are randomly selected from the field set (two fields are selected in this embodiment);

[0077] 1.4) Record the randomly selected fields in the storage medium;

[0078] 1.5) Replace the value of the variable name agreed in step 1.2) with the value of the field variable name obtained in step 1.3);

[0079] 1.6) Complete the subsequent project construction.

[0080] After the front-end static resources are published, it is necessary to color the interface requests during normal user use, namely:

[0081] 2) Coloring user interface requests, such as Figure 3 As shown, the specific steps include:

[0082] 2.1) Get the field to be encrypted from the encrypted script;

[0083] 2.2) Pre-read the value of the encrypted field in the request and compose it into a string. For example, the string can be composed in the form of [field 1=xxx&field 2=xxx];

[0084] 2.3) Encrypt the composed string. In this embodiment, the MD5 encryption algorithm is selected, which has good compressibility, is easy to calculate, has strong resistance to modification and collision, and has a powerful algorithm and high cracking difficulty;

[0085] 2.4) Add the key value to the request header and assign it the encrypted value obtained in step 2.3);

[0086] 2.5) Send an interface request to the gateway.

[0087] 3) After the request is sent, the interface is transmitted through the network to the gateway, and the gateway analyzes, judges and marks the user's interface request, and decides whether to release or reject the user's interface request based on the marking result; the specific steps include: Figure 4 As shown:

[0088] 3.1) The gateway intercepts the interface request and reads the encrypted value of the interface request;

[0089] 3.2) Compose the encrypted value into a string. Similarly, the string can be composed in the form of [field 1=xxx&field 2=xxx];

[0090] 3.3) Encrypt the string. Correspondingly, the MD5 encryption algorithm is also used in this embodiment;

[0091] 3.4) Read the key value agreed in the Header value in the interface request;

[0092] 3.5) Determine whether the encrypted values ​​obtained in step 3.3) and step 3.4) are equal, and perform corresponding color marking;

[0093] When the judgment results are equal, the interface request is marked with a color;

[0094] When the judgment results are not equal, the interface request is not marked.

[0095] 3.6) Determine whether to release the interface request based on the staining result obtained in step 3.5), that is:

[0096] When the interface request has a dye mark, the interface request is allowed to be released;

[0097] When the interface request does not have a dye mark, the interface request is rejected.

[0098] 3.7) Release or reject the interface request accordingly.

[0099] In order to further improve the encryption and reduce the possibility of being cracked, a series of irregular strings can be generated as the script file name using a random sequence each time the script is built. The content in the script file includes an obfuscated script and an encrypted script of the same size. After the construction is completed, the corresponding encrypted scripts will be hidden in different script files. That is, the embodiment of the present invention also includes the steps of:

[0100] 4) Construct static resource interference traps, such as Figure 5 As shown, the specific steps include:

[0101] 4.1) Repeat the steps in step 1) to build the front-end static resources;

[0102] 4.2) Generate several sets of random strings using openssl or shell scripts;

[0103] 4.3) Change the file names of the puppet script and the encrypted script to the file names of the random strings generated in step 4.2);

[0104] 4.4) Update the mapping relationship between script files and file names.

[0105] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

Claims

1. A method for identifying and defending against trusted crawlers, characterized in that: The following steps are involved: 1) Build front-end static resources; 2) Coloring user interface requests; 3) The gateway analyzes, judges and marks the user's interface request, and decides whether to release or reject the user's interface request based on the marking result; 4) Construct static resource interference traps; The step 1) includes the following steps: 1.1) Initialize the set of encryptable fields; 1.2) Agree on the file names and variable names of the encrypted files and the encrypted fields defined in the encrypted files; 1.3) Randomly select several fields from the field set in step 1.1); 1.4) Record the randomly selected fields in the storage medium; 1.5) Replace the value of the variable name agreed in step 1.2) with the value of the field variable name obtained in step 1.3); The step 1.1) includes the following steps: 1.11) Analyze all request interfaces or sensitive interfaces of the system; 1.12) Extract the common fields in the analyzed interface; 1.13) Store the interfaces that meet the preset conditions in the storage medium in the form of a list; The step 2) includes the following steps: 2.1) Get the field to be encrypted from the encrypted script; 2.2) Pre-read the value of the encrypted field in the request and compose it into a string; 2.3) Encrypt the composed string; 2.4) Add the key value to the request header and assign it the encrypted value obtained in step 2.3); 2.5) Send an interface request to the gateway; The step 3) includes the following steps: 3.1) The gateway intercepts the interface request and reads the encrypted value of the interface request; 3.2) Compose the encrypted value into a string; 3.3) Encrypt the string; 3.4) Read the key value agreed in the Header value in the interface request; 3.5) Determine whether the encrypted values ​​obtained in step 3.3) and step 3.4) are equal, and perform corresponding coloring marks; 3.6) Determine whether to release the interface request according to the staining result obtained in step 3.5); 3.7) Release or reject the interface request accordingly; The above 4) comprises the following steps: 4.1) Repeat the steps in step 1) to build the front-end static resources; 4.2) Generate several sets of random strings; 4.3) Change the file names of the puppet script and the encrypted script to the file names of the random strings generated in step 4.2); 4.4) Update the mapping relationship between script files and file names.

2. A trusted crawler identification and defense method according to claim 1, characterized in that: The storage medium is a database or a file.

3. A trusted crawler identification and defense method according to claim 1, characterized in that: In step 3.5), When the judgment results are equal, the interface request is marked with a color; When the judgment results are not equal, the interface request is not marked.

4. A trusted crawler identification and defense method according to claim 1, characterized in that: In step 3.6), When the interface request has a dye mark, the interface request is allowed to be released; When the interface request does not have a dye mark, the interface request is rejected.

5. A trusted crawler identification and defense method according to claim 1, characterized in that: In 4.2), a random string is generated using openssl or a shell script.