Crawler identification protection method based on honeypot network security technology

Abstract

The invention discloses a crawler identification protection method based on a honeypot network security technology, and particularly relates to the technical field of network security protection. The method comprises the following steps: setting invisible honeypot entries in a webpage code form and randomly extracted visible honeypot posts in a webpage information list, and in combination with verification and identification of post serial numbers in an access request, accurately determining crawling behaviors of an access user on different types of honeypot posts; the network access address and the user agent identification information of the access user are further extracted and matched with the preset search engine user agent white list, so that the non-search engine crawler access user is automatically identified, and the non-search engine crawler access source is effectively forbidden for preset duration. According to the invention, crawler identification accuracy and protection intensity can be effectively improved, the risk of wrong sealing is reduced, and network security protection efficiency is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the network security protection technical field, more particularly, the present application relates to a crawler identification protection method based on honeypot network security technology. BACKGROUND

[0002] In the existing webpage information protection technology, generally, by adding confusion elements or limiting access frequency in the webpage source code or front-end display, etc., it is attempted to identify or limit the automatic crawling of webpage data by the crawler user. The crawler user usually adopts the mode of simulating normal user browsing behavior or customizing rules to parse the webpage source code to bypass such basic protection strategy, so that the website cannot effectively distinguish the real user from the automatic crawling tool, thereby causing network security problems such as data leakage, waste of server resources, and decline of service quality. Therefore, the technical problem existing in the prior art is: how to accurately distinguish the normal user from the automatic webpage crawler user, so as to avoid the website data from being massively crawled by the unauthorized automatic tool. SUMMARY

[0003] In order to overcome the above-mentioned defects of the prior art, the embodiments of the present application provide a crawler identification protection method based on honeypot network security technology to solve the problems proposed in the background art.

[0004] In order to achieve the above-mentioned purpose, the present application provides the following technical scheme: A crawler identification protection method based on honeypot network security technology, comprising the following steps: S1: selecting a predetermined serial number entry in the webpage information list, setting the CSS style as hidden, to generate a non-visible honeypot entry in the form of webpage code, and randomly extracting part of the serial numbers from the continuous webpage post serial numbers as visible honeypot posts; S2: when the access user accesses the webpage, the post serial number in the access request link is obtained, and the honeypot post data table is queried, whether the post serial number is located in the honeypot post data table is judged, to determine whether the access user crawls the non-visible honeypot entry in the form of webpage code; S3: when the access user accesses the webpage, the post serial number in the access request link is obtained, and the honeypot post data table is queried, whether the post serial number is located in the honeypot post data table is judged, to determine whether the access user crawls the randomly extracted visible honeypot post; S4: for the access user who has judged to crawl the non-visible honeypot entry in the form of webpage code and the randomly extracted visible honeypot post, the corresponding network access address and user agent identification information are extracted; S5: Based on the user agent identification information of the access user and the preset search engine user agent white list, it is judged whether the access user belongs to the non-search engine crawler, and the network access address of the non-search engine crawler is blocked for a preset time length.

[0005] In a preferred embodiment, S1, specifically: Extracting consecutive web post serial numbers in the web information list to form a candidate serial number set; Randomly selecting a honeypot post serial number from the candidate serial number set to form a honeypot post serial number set and writing it into a honeypot post data table; Selecting a predetermined serial number item from the honeypot post serial number set to form a predetermined serial number item set; Setting the CSS style of the web information list item corresponding to the predetermined serial number item set to hidden and writing it into the web information list web code to generate a web code form invisible honeypot item, and setting the visible honeypot post according to the honeypot post serial number set.

[0006] In a preferred embodiment, S2, specifically: Extracting the post serial number in the access request link of the web access request initiated by the access user to form a request post serial number; Performing set membership verification on the request post serial number and the honeypot post serial number set in the honeypot post data table to generate a membership verification result; When the membership verification result indicates that the request post serial number is located in the honeypot post serial number set, it is determined that the access user crawls the web code form invisible honeypot item.

[0007] In a preferred embodiment, S3, specifically: Extracting the post serial number in the access request link of the web access request initiated by the access user to form a request post serial number; Performing set membership verification on the request post serial number and the honeypot post serial number set in the honeypot post data table to generate a membership verification result; When the membership verification result indicates that the request post serial number is located in the honeypot post serial number set and the request post serial number is not located in the predetermined serial number item set, it is determined that the access user crawls the randomly extracted visible honeypot post.

[0008] In a preferred embodiment, S4, specifically: Generating a honeypot hit access user set by summarizing the results of determining that the access user crawls the web code form invisible honeypot item and determining that the access user crawls the randomly extracted visible honeypot post; Based on the honeypot hit access user set, extracting the web access request corresponding to the honeypot hit access user set; Reading the network access address and reading the user agent identification information based on the webpage access request; The network access address and the user agent identification information are associated and written into the data record corresponding to the honeypot hit access user set to form the corresponding relationship between the network access address and the user agent identification information.

[0009] In a preferred embodiment, S5, specifically: Based on the corresponding relationship between the network access address and the user agent identification information, the user agent identification information is extracted and matched with the preset search engine user agent whitelist to generate a whitelist matching result; Based on the whitelist matching result, the network access address is filtered to form a non-search engine crawler network access address set; The non-search engine crawler network access address set is executed for a preset duration ban, and the network access address, the ban start time and the ban termination condition are written into the network access address ban record.

[0010] In a preferred embodiment, based on the corresponding relationship between the network access address and the user agent identification information, the user agent identification information set is extracted; Based on the user agent identification information set, the network access address ban record is summarized to generate user agent identification information access frequency data; Based on the user agent identification information access frequency data, high-frequency user agent identification information is filtered to form a high-frequency user agent identification information set; The high-frequency user agent identification information set and the user agent identification information access frequency data are submitted to a preset prompt platform to generate a manual review task queue.

[0011] In a preferred embodiment, based on the network access address ban record, network access address prefix information is extracted to generate a network access address segment candidate set; The network access address segment candidate set is executed for ban frequency statistics to generate network access address segment frequency data; Based on the network access address segment frequency data, high-frequency network access address segments are filtered to form a high-frequency network access address segment set; The high-frequency network access address segment set is written into the network access address segment ban rule and associated with the network access address ban record.

[0012] The technical effect and advantages of the crawler identification protection method based on the honeypot network security technology: The invisible honeypot entry in the form of webpage code is set in the webpage information list, and the visible honeypot post is configured, so that when the access request reaches the honeypot, verifiable abnormal access evidence is formed; the post serial number in the access request link is extracted and set membership verification is performed on the honeypot post data table, so as to identify the access behavior of the invisible honeypot entry in the form of webpage code and the access behavior of the randomly extracted visible honeypot post, respectively, and the coverage ability of different crawling modes is improved; the network access address and user agent identification information corresponding to the honeypot hit access user are extracted, so as to form traceable source identification and behavior record; the user agent identification information is matched with the preset search engine user agent white list, so as to realize the distinction between search engine access and non-search engine crawler access; and the network access address of the non-search engine crawler is implemented for preset time length ban, so as to enhance the blocking ability and consistency of disposal of unauthorized crawling without affecting the normal crawling of the search engine. BRIEF DESCRIPTION OF DRAWINGS

[0013] Figure 1 A schematic diagram of a crawler identification and protection method based on a honeypot network security technology is given. DETAILED DESCRIPTION

[0014] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0015] EMBODIMENT Figure 1 A crawler identification and protection method based on a honeypot network security technology is given, which comprises the following steps: S1: selecting a predetermined serial number entry in the webpage information list, setting the CSS style as hidden to generate an invisible honeypot entry in the form of webpage code, and randomly extracting part of the serial numbers from the continuous webpage post serial numbers as visible honeypot posts; S2: when an access user accesses a webpage, the post serial number in the access request link is obtained, and the honeypot post data table is queried to determine whether the access user crawls the invisible honeypot entry in the form of webpage code by judging whether the post serial number is located in the honeypot post data table; S3: when an access user accesses a webpage, the post serial number in the access request link is obtained, and the honeypot post data table is queried to determine whether the access user crawls the randomly extracted visible honeypot post by judging whether the post serial number is located in the honeypot post data table; S4: Extract the corresponding network access address and user agent identification information of the access user of the invisible honeypot item in the form of the crawled webpage code and the randomly extracted visible honeypot post; S5: Match the user agent identification information of the access user with the preset search engine user agent white list to determine whether the access user belongs to a non-search engine crawler, and block the network access address of the non-search engine crawler for a preset time length.

[0016] S1: Select a predetermined serial number item in the webpage information list, set the CSS style to hidden to generate an invisible honeypot item in the form of webpage code, and randomly extract part of the serial numbers from the continuous webpage post serial numbers as visible honeypot posts, including: Extracting continuous webpage post serial numbers from the webpage information list to form a candidate serial number set; Randomly selecting honeypot post serial numbers from the candidate serial number set to form a honeypot post serial number set and writing into a honeypot post data table; Selecting a predetermined serial number item from the honeypot post serial number set to form a predetermined serial number item set; Setting the CSS style of the webpage information list item corresponding to the predetermined serial number item set to hidden and writing into the webpage information list webpage code to generate an invisible honeypot item in the form of webpage code, and setting visible honeypot posts according to the honeypot post serial number set.

[0017] In the embodiment of extracting continuous web post serial numbers to form a candidate serial number set from a web information list, the web information list points to a set of web entries displayed to a visiting user, each web entry in the set of web entries contains a post serial number field, a post title field, and a visiting request link field, the visiting request link field contains a post serial number parameter corresponding to the post serial number field, the post serial number parameter is used to locate a post content page in a visiting request, the post serial number field is stored in a comparable integer form and remains unique within the web information list, the continuous web post serial numbers refer to a sequence of post serial numbers in the web information list that satisfy an increasing order relationship and have a difference of 1 between adjacent serial numbers, the candidate serial number set refers to a set of post serial numbers for honeypot post serial number extraction and is stored in units of post serial numbers after deduplication, the extraction process of continuous web post serial numbers includes reading the post serial number field from the web information list one by one and sorting them by numerical value, after the sorting result forms a sequence of post serial numbers, the difference between adjacent serial numbers is determined, the adjacent serial numbers with a difference of 1 belong to the same continuous section, the adjacent serial numbers with a difference not equal to 1 are divided into different continuous sections, the continuous section selection rule adopts the principle of giving priority to the maximum length of the continuous section or the principle of giving priority to the maximum number of web information list entries covered by the continuous section, the continuous section selection rule is used to avoid the sparseness of the candidate serial number set caused by multiple short continuous sections in the web information list, after the selection of the continuous section is completed, all post serial numbers in the continuous section are written into the candidate serial number set and the generation time stamp of the candidate serial number set is recorded for traceability.

[0018] In the implementation of randomly selecting the honeypot post serial numbers from the candidate serial number set to form the honeypot post serial number set and writing into the honeypot post data table, the random selection refers to unbiased extraction within the element space of the candidate serial number set and ensuring that the extraction result is within the range of the candidate serial number set, the honeypot post serial number refers to a post serial number used to induce network crawler access and as a recognition trigger condition, the honeypot post serial number set refers to a set of honeypot post serial numbers and is stored in the honeypot post data table as a set field or multiple rows of records, the honeypot post data table refers to a structured data table for storing the honeypot post serial number set and supporting set attribution verification, the structured data table includes a honeypot post serial number field, a honeypot post type field, a creation timestamp field, and a valid state field, the honeypot post type field is used to distinguish between invisible honeypot entries associated with honeypot post serial numbers in the form of web page code and visible honeypot posts associated with honeypot post serial numbers, the random selection process adopts extraction quantity determination rules and extraction execution rules, the extraction quantity determination rules are based on the size of the candidate serial number set for grading setting and output as an integer quantity, the grading setting method adopts a candidate serial number set size interval mapping method and ensures that the extraction quantity is less than the size of the candidate serial number set, the extraction execution rules adopt non-replacement extraction and record the extracted post serial numbers through an extraction marker set to avoid repeated extraction, the extraction execution rules can adopt a random index extraction method based on uniform distribution, the input of the random index extraction method is the ordered array representation of the candidate serial number set and a random seed, the random seed is obtained from the combination of the current timestamp of the server and the session identifier in the request context and is compressed by a hash function to obtain a fixed-length random seed, the random index extraction method outputs an extraction index sequence and maps to obtain the honeypot post serial number set, after the honeypot post serial number set is generated, the honeypot post serial number set is written into the honeypot post data table row by row and written into the creation timestamp field and the valid state field, the writing operation adopts a transaction submission method to ensure the atomicity of the writing of the honeypot post serial number set.

[0019] In the embodiment of selecting predetermined serial number items from the honey post serial number set to form the predetermined serial number item set, the predetermined serial number item refers to the post serial number selected from the honey post serial number set and used to generate the invisible honey item in the form of web page code, the predetermined serial number item set refers to the set of all predetermined serial number items and used to locate the web information list item in the hidden writing process, the predetermined serial number item selection rule adopts the combination of the position dispersion rule and the page semantic consistency rule, the position dispersion rule refers to keeping the predetermined serial number items apart in the sorting position of the honey post serial number set to avoid concentrating in a single section of the candidate serial number set, the implementation of the position dispersion rule is to sample the honey post serial number set at equal intervals after sorting by numerical value and perform availability verification near the sampling point, the availability verification refers to checking whether the post serial number exists in the web information list corresponding to the access request link field and the access request link field format meets the post serial number parameter parsing condition, the page semantic consistency rule refers to keeping the predetermined serial number item corresponding to the web information list item consistent with the ordinary web information list item in the structure field, including the title field length range, the link field prefix structure and the display field arrangement order, the page semantic consistency rule is used to ensure that the invisible honey item in the form of web page code does not introduce abnormal structure marks in the web information list web page code, after forming the predetermined serial number item set, the predetermined serial number item set is written into the honey post type field of the honey post data table to mark the association relationship of the invisible honey item in the form of web page code, and the predetermined serial number item set is cached in the page generation context in the form of a set field so as to write the web information list web page code.

[0020] In the implementation of setting the CSS style of the webpage information list item corresponding to the predetermined serial number item set to hidden and writing the invisible honeypot item in the form of webpage code into the webpage information list webpage code, the webpage information list webpage code refers to the hypertext markup language code text generated by the server side and returned to the visiting user, the invisible honeypot item in the form of webpage code refers to the item existing in the webpage information list webpage code and adopting the same item structure tag as the normal item of the webpage information list but invisible at the rendering level, the CSS style of hidden refers to the style state of making the item element not participate in the visual layout of the page or participating in the layout but invisible through the Cascading Style Sheets property setting, the CSS style of hidden can adopt the combination mode of setting the display property to none or the visibility property to hidden or the opacity property to 0 and combining the height and width to zero, the item structure tag of the webpage information list refers to the container tag for carrying the item content and including the data attribute corresponding to the post serial number field and the hyperlink tag corresponding to the access request link field, the hidden writing process includes item positioning, style injection and code splicing, the item positioning is completed by traversing the webpage information list item and matching the post serial number field with the predetermined serial number item set, the matching mode adopts the set attribution verification and takes the post serial number field as the key, the style injection appends the inline style attribute in the item structure tag or appends the class name and writes the CSS style of hidden defined for the corresponding class name in the webpage header stylesheet, the code splicing splices the item structure tag with the completed style injection and the other item structure tags of the webpage information list according to the original order to generate the webpage information list webpage code, after the generation of the webpage information list webpage code, a browser rendering consistency verification is performed, the browser rendering consistency verification refers to the verification of the number of item structure tags, the closure relationship of item structure tags and the integrity of the access request link field in the server side template rendering stage and ensures that the webpage information list webpage code still contains the access request link field corresponding to the predetermined serial number item set.

[0021] In the implementation of setting the visible honeypot post according to the honeypot post serial number set, the visible honeypot post refers to a web information list item that is presented in the normal visible display area of the web information list and whose access request link field points to a honeypot post serial number parameter in the honeypot post serial number set. The setting process of the visible honeypot post includes visible item mapping and visible item publishing. The visible item mapping refers to mapping the honeypot post serial number in the honeypot post serial number set to the content source of the web information list item. The mapping method adopts post serial number field replacement or new item insertion while keeping the web information list item structure label consistent. The post serial number field replacement refers to updating the post serial number field in the item structure label to the honeypot post serial number and synchronously updating the post serial number parameter in the access request link field without changing the item structure label. The new item insertion refers to inserting a new item with the same structure as the normal item in the web information list item sequence and pointing the post serial number field of the new item and the access request link field to the honeypot post serial number in the honeypot post serial number set. The insertion position selection rule of the new item insertion adopts the same insertion principle as the web information list sorting rule to avoid the visible honeypot post being conspicuous. The web information list sorting rule can adopt publishing time sorting or post serial number sorting and keep the sorting stability during insertion. The visible item publishing refers to publishing the web information list web code after mapping to the external service and ensuring that the access request link field can be clicked by the access user to trigger the web access request.

[0022] The data consistency maintenance method of the honeypot post data table in the whole process includes idempotency constraint of honeypot post data table writing and updating and lifecycle management of the valid state field. The idempotency constraint refers to avoiding repeated insertion when the same honeypot post serial number is in the valid state field. The implementation of the idempotency constraint is to establish a uniqueness check on the honeypot post serial number field and perform an existence query before writing. The lifecycle management of the valid state field refers to invalidating the honeypot post serial number set when the candidate serial number set is updated or the web information list is updated and generating a new honeypot post serial number set. The invalidation is completed by updating the valid state field of the corresponding record in the honeypot post data table to invalid and writing the invalidation timestamp field. After the writing of the new honeypot post serial number set is completed, the pre-determined serial number item set reselection is performed and the web information list web code regeneration is performed. The trigger condition of the web information list web code regeneration is provided by the web information list update event or the timing refresh event. The web information list update event refers to the change of the number of web information list items or the change of the post serial number distribution. The timing refresh event refers to regenerating the candidate serial number set and the honeypot post serial number set at a fixed period and covering the old set.

[0023] The uniformity of the invisible honeypot entry in the form of webpage code and the visible honeypot post at the data association level is achieved through a honeypot post data table. The honeypot post type field in the honeypot post data table is used to mark the honeypot post serial number associated with the invisible honeypot entry in the form of webpage code and the visible honeypot post associated with the honeypot post serial number. The intersection relationship between the pre-determined serial number entry set and the honeypot post serial number set is explicitly established through a pre-determined serial number entry set selection process and written into the honeypot post data table. The webpage information list webpage code generation process reads the pre-determined serial number entry set to locate the hidden entry and reads the honeypot post serial number set to locate the visible entry mapping. The honeypot post data table provides set attribution verification basis in the access request processing and ensures that the request post serial number can be directly executed with the honeypot post serial number set for set attribution verification, completing the closed-loop data link from the webpage information list construction to the access request identification.

[0024] S2: When the access user accesses the webpage, the post serial number in the access request link is obtained, and the honeypot post data table is queried to determine whether the access user crawls the invisible honeypot entry in the form of webpage code by judging whether the post serial number is located in the honeypot post data table, including: The post serial number in the access request link is extracted from the webpage access request initiated by the access user to form a request post serial number; The request post serial number is executed with the honeypot post serial number set in the honeypot post data table to generate an attribution verification result; When the attribution verification result represents that the request post serial number is located in the honeypot post serial number set, it is determined that the access user crawls the invisible honeypot entry in the form of webpage code.

[0025] Specifically, in the implementation of extracting the post serial number in the access request link to form the request post serial number for the web access request initiated by the access user, the web access request points to the hypertext transfer protocol request message submitted by the server side, the hypertext transfer protocol request message includes a request line, a request header field and a request body field, the request line carries a uniform resource locator path string, the uniform resource locator path string includes the post serial number parameter corresponding to the access request link field, and the post serial number parameter is encoded in the query string section of the uniform resource locator path string in the form of a key-value pair. After the uniform resource locator path string enters the parsing process, the parsing process first performs separator splitting on the query string section, the separator includes a question mark character and a connection character, and the key-value pair sequence obtained by splitting enters a key matching process, the key matching process takes the post serial number parameter key name as a matching condition and locates the corresponding key value, and the key value located is a post serial number string. After the post serial number string enters the normalization process, blank character elimination and non-digit character checking are performed, the non-digit character checking is performed in a character-by-character traversal manner and takes the character code falling within the range of digit characters as the basis for judgment, and when the non-digit character checking fails, the post serial number string is marked as an invalid post serial number string and the set membership checking is terminated. After the post serial number string completes the non-digit character checking, it enters the numerical conversion process, the numerical conversion process converts the post serial number string into an integer and forms a request post serial number, the request post serial number is used as input data for set membership checking and maintains the same request context identifier as the writing process of the network access address and user agent identifier information to support tracing.

[0026] In the implementation of performing set membership check on the request post serial number and the honeypot post serial number set in the honeypot post data table to generate a membership check result, the set membership check refers to a determination process of determining whether the request post serial number belongs to the honeypot post serial number set. The honeypot post serial number set is expressed by a set of honeypot post serial number fields with valid status fields in the honeypot post data table, and is distinguished by honeypot post type fields to associate honeypot post serial numbers of invisible honeypot entries in webpage code form with honeypot post serial numbers of visible honeypot posts. The execution process of the set membership check first generates a set of data table query conditions, which at least includes a honeypot post serial number field equal to the request post serial number, a valid status field being valid, and a honeypot post type field being the association type of invisible honeypot entries in webpage code form. The set of data table query conditions enters a data table query statement generation process and outputs a data table query statement. The data table query statement is executed on the honeypot post data table to output a set of query result records. When the set of query result records is empty, the membership check result is generated as a miss. When the set of query result records is not empty, the membership check result is generated as a hit and the creation timestamp field and the honeypot post type field in the set of query result records are extracted and written into a membership check result record. The membership check result record is stored in a structured record form, which includes a request post serial number field, a membership check result field, a query execution timestamp field, and a query condition set field. The membership check result field takes a value of hit or miss. The query condition set field is used to reproduce the same determination path after the honeypot post data table is updated and to maintain data consistency.

[0027] In the implementation of determining the access user's crawling of invisible honeypot entries in the form of webpage code, where the attribution verification result indicates that the request post number is located in the honeypot post number set, determining the access user's crawling of invisible honeypot entries in the form of webpage code involves establishing an explicit association between the access user and the access behavior of invisible honeypot entries in the form of webpage code and outputting a crawling judgment record. The crawling judgment record generation process takes the attribution verification result record as input and reads the attribution verification result field. When the attribution verification result field is a match, the crawling judgment record writing process begins. The crawling judgment record writing process combines the request post number, the access user identifier, and the honeypot post type field from the honeypot post data table query result record set into the crawling judgment record. The access user identifier is generated using a key construction method that maps network access addresses to user agent identifier information. This key construction method concatenates the network access address and user agent identifier information using a fixed connector to form an access user identifier string and performs a length limit check on the access user identifier string to ensure storage consistency. The crawling judgment record stores the following fields: access user identifier, request post sequence number, honeypot post type, attribution verification result, and judgment timestamp. The judgment timestamp is taken from the server's current time and has a one-to-one correspondence with the query execution timestamp of the attribution verification result record. The attribution verification result field generates a negative branch record for crawling judgment records when a match is found. The negative branch record contains the access user identifier, request post sequence number, attribution verification result, and judgment timestamp and is used for baseline sample management in statistical analysis.

[0028] To ensure the stability of the determination path for invisible honeypot entries in webpage code format under updated honeypot post data table conditions, the set attribution verification also includes consistency checks for the valid status field and the honeypot post type field. The valid status field consistency check means that if the attribution verification result is a hit, the valid status field of the record corresponding to the same honeypot post sequence number field in the honeypot post data table is read again and determined to be valid. The honeypot post type field consistency check means that if the attribution verification result is a hit, the honeypot post type field is read again and determined to be still the association type of the invisible honeypot entry in webpage code format. If the valid status field consistency check fails, the attribution verification result field is updated to "miss," and the consistency check failure reason field is recorded. The value of the consistency check failure reason field is limited to either a change in the valid status field or a change in the honeypot post type field, and is consistent with the audit. After the honeypot post data table query result record set passes the consistency check, it enters the crawling judgment record writing process and completes the closed loop of judging invisible honeypot entries in the form of crawled web page code of the accessing user. The judgment closed loop consists of request post sequence number extraction, set ownership verification, ownership verification result record generation, accessing user identifier generation and crawling judgment record writing, and ensures that the crawling judgment record can be directly read as input data.

[0029] S3: When a user accesses a webpage, retrieve the post number from the access request link and query the honeypot post data table. By determining whether the post number is present in the honeypot post data table, it is possible to ascertain whether the user has crawled randomly selected visible honeypot posts, including: Extract the post sequence number from the webpage access request link initiated by the user to form the request post sequence number; Perform a set ownership check on the request post sequence number and the honeypot post sequence number set in the honeypot post data table to generate an ownership check result; When the attribution verification result indicates that the requested post sequence number is in the honeypot post sequence number set but not in the predetermined sequence number entry set, it is determined that the accessing user crawled randomly selected visible honeypot posts.

[0030] Specifically, in the implementation of extracting the post sequence number from the access request link in a webpage access request initiated by a user to form the request post sequence number, the webpage access request is a Hypertext Transfer Protocol (HTTP) request message received by the server. The HTTP request message includes a request line and a request header field. The request line carries a Uniform Resource Locator (URI) path string, which contains the post sequence number parameter corresponding to the access request link field. The post sequence number parameter is encoded as a key-value pair in the query string segment of the URI path string. After the URI path string enters the parsing process, the query string segment is split by a delimiter to obtain a key-value pair sequence. The key-value pair sequence enters the key matching process and uses the post sequence number parameter as the matching condition to locate the key value. The located key value is used as the post sequence number string to enter the normalization process and performs whitespace character removal and non-numeric character verification. The non-numeric character verification adopts a character-by-character traversal and uses the character encoding falling within the range of numeric characters as the judgment criterion. After the non-numeric character verification passes, the post sequence number string is numerically converted to form the request post sequence number.

[0031] In the implementation method of generating an attribution verification result by performing set attribution verification on the request post sequence number and the honeypot post sequence number set in the honeypot post data table, the honeypot post data table is a structured data table used to store the honeypot post sequence number set. The structured data table includes a honeypot post sequence number field, a honeypot post type field, a creation timestamp field, and a validity status field. The honeypot post sequence number set consists of honeypot post sequence numbers whose validity status field is valid and is used for set attribution verification. The set attribution verification takes the request post sequence number as input and generates a set of data table query conditions. The set of data table query conditions must at least include the honeypot post sequence number field being equal to the request post sequence number and the validity status field being valid. The set of data table query conditions enters the data table query statement generation process and outputs the data table query statement. After the data table query statement is executed on the honeypot post data table, it outputs a set of query result records. When the set of query result records is empty, an attribution verification result of "miss" is generated and written to the attribution verification result record. When the set of query result records is not empty, an attribution verification result of "hit" is generated and the honeypot post type field and creation timestamp field from the set of query result records are extracted and written to the attribution verification result record.

[0032] In the implementation of the pre-determined sequence number entry set, the pre-determined sequence number entry set is a set of post sequence numbers selected from the honeypot post sequence number set and used to generate invisible honeypot entries in the form of web page code. The pre-determined sequence number entry set is cached in the page generation context as a set field and synchronously written into the honeypot post data table as a set of records of the invisible honeypot entry association type in the form of web page code. The pre-determined sequence number entry set enters the loading process during the access request processing stage. The loading process first reads the pre-determined sequence number entry set from the page generation context, and if the page generation context is missing, it generates the pre-determined sequence number entry set from the honeypot post data table according to the invisible honeypot entry association type in the form of web page code for the honeypot post type field and the valid status field as valid. After the pre-determined sequence number entry set is loaded, it enters the set ownership verification preparation process and uses the request post sequence number as the comparison key.

[0033] In the implementation where the attribution verification result indicates that the request post sequence number is in the honeypot post sequence number set but not in the predetermined sequence number entry set, the determination that the request post sequence number is not in the predetermined sequence number entry set adopts a set exclusion verification process. The set exclusion verification process takes the request post sequence number and the predetermined sequence number entry set as input and outputs the exclusion verification result. The exclusion verification result is either pass or fail. A pass result indicates that the request post sequence number is not in the predetermined sequence number entry set, and a fail result indicates that the request post sequence number is in the predetermined sequence number entry set. The set exclusion verification process can use a hash set query method to improve processing efficiency. The hash set query method converts the predetermined sequence number entry set into a hash set and performs an existence query using the request post sequence number as the key to generate the exclusion verification result.

[0034] In the implementation of determining the access user to crawl randomly selected visual honeypot posts, when the attribution verification result is a hit and the exclusion verification result is passed, the visual honeypot post crawling judgment record generation process is entered. The visual honeypot post crawling judgment record generation process reads the request post sequence number and reads the honeypot post type field of the record corresponding to the request post sequence number in the honeypot post data table. When the honeypot post type field is a visual honeypot post associated type, it is written into the visual honeypot post crawling judgment record. The visual honeypot post crawling judgment record includes an access user identifier field, a request post sequence number field, an attribution verification result field, an exclusion verification result field, a honeypot post type field, and a judgment timestamp field. The access user identifier field is generated by using a key construction method that corresponds to the network access address and user agent identifier information. The key construction method concatenates the network access address and user agent identifier information with a fixed connector to form an access user identifier string and performs a length limit verification on the access user identifier string to maintain storage consistency. The judgment timestamp field is taken from the current time of the server and establishes a correspondence with the query execution timestamp field of the attribution verification result record.

[0035] When the attribution verification result is a hit and the exclusion verification result is a failure, the invisible honeypot entry path marking process is initiated. The invisible honeypot entry path marking process writes the request post sequence number into the invisible honeypot entry path marking record and associates and stores the invisible honeypot entry path marking record with the attribution verification result record. When the attribution verification result is a miss, the non-honeypot access path marking process is initiated and a non-honeypot access path marking record is generated. The non-honeypot access path marking record contains the access user identifier field, the request post sequence number field, the attribution verification result field, and the judgment timestamp field, and is used for baseline sample management in statistical analysis.

[0036] To ensure the stability of the visible honeypot post determination path under updated honeypot post data table conditions, the set attribution verification and set exclusion verification also include valid status field consistency verification and pre-determined sequence number item set consistency verification. When the attribution verification result is a hit, the valid status field of the record corresponding to the request post sequence number in the honeypot post data table is read again to determine that the valid status field is still valid. When the exclusion verification result is a pass, the honeypot post type field of the record corresponding to the request post sequence number in the honeypot post data table is read again to determine that the honeypot post type field is still a visible honeypot post association type. When the consistency verification fails, the visible honeypot post crawling determination record generation process is rolled back to the non-honeypot access path marking process and the consistency verification failure reason field is written to maintain audit consistency.

[0037] S4: For users accessing invisible honeypot entries whose webpage code format has been identified and randomly selected visible honeypot posts, extract the corresponding network access addresses and user agent identification information, including: The results of crawling invisible honeypot entries in the form of web page code by the determined users are combined with the results of crawling randomly selected visible honeypot posts by the determined users to generate a set of honeypot-hit users. Extract webpage access requests corresponding to the honeypot-hit user set based on the honeypot-hit user set; Based on the web page access request, read the network access address and read the user agent identification information; The network access address is associated with the user agent identification information and written into the data record corresponding to the set of users accessing the honeypot, thus forming a correspondence between the network access address and the user agent identification information.

[0038] Specifically, in the implementation of generating a honeypot-hit user set by summarizing the results of unseen honeypot entries in the form of webpage code crawled by the accessing user and the results of randomly selected visible honeypot posts crawled by the accessing user, the results of unseen honeypot entries in the form of webpage code crawled by the accessing user and the results of randomly selected visible honeypot posts crawled by the accessing user are uniformly stored in the form of judgment result records. The judgment result record includes a request post sequence number field, an attribution verification result field, an exclusion verification result field, a judgment timestamp field, and an access request identifier field. The access request identifier field is used to uniquely locate the same webpage access request and form an alignment key across records. The access request identifier field is generated by using the concatenation string of the Uniform Resource Locator (URL) path string and the judgment timestamp field as input and performing a hash function operation to obtain a fixed-length identifier string. The fixed-length identifier string output by the hash function operation is written into the access request identifier field and written into the judgment result record storage medium.

[0039] The process of generating the honeypot hit access user set takes the judgment result records as input and performs hit filtering and set merging. The hit filtering rules limit the value of the attribution verification result field to "hit" and the value of the exclusion verification result field to meet the judgment conditions of visible honeypot posts or invisible honeypot entries. The judgment conditions of invisible honeypot entries are the combination of the attribution verification result field being "hit" and the exclusion verification result field not participating in the judgment. The judgment conditions of visible honeypot posts are the combination of the attribution verification result field being "hit" and the exclusion verification result field being "pass". The hit filtering outputs the hit judgment result record set and performs set merging with the access request identifier field as the deduplication key. The set merging process reads the access request identifier field of each record in the hit judgment result record set and writes it into the honeypot hit access user set. The honeypot hit access user set is stored in a set data structure and contains the access request identifier field set and the request post sequence number field set. The set data structure performs an existence query of the access request identifier field during the writing phase to avoid duplicate writing and ensure the uniqueness of the honeypot hit access user set.

[0040] In the implementation of extracting webpage access requests corresponding to a honeypot-hit user set, the webpage access requests are stored in the form of webpage access request log records. These log records include an access request identifier field, a Uniform Resource Locator (URL) path string field, a set of request header fields, a receive timestamp field, and a response status field. The webpage access request log records are generated on the server side when receiving a Hypertext Transfer Protocol (HTTP) request message and share the access request identifier field with the judgment result record. The extraction process takes the set of access request identifier fields from the honeypot-hit user set as input and performs an equality matching query on the webpage access request log records. The equality matching query condition requires that the access request identifier field of the webpage access request log record is equal to the access request identifier field of the honeypot-hit user set. The equality matching query outputs the honeypot-hit webpage access request set while maintaining a one-to-one correspondence between each webpage access request log record in the honeypot-hit webpage access request set and the access request identifier field in the honeypot-hit user set.

[0041] In this implementation method based on reading network access addresses and user agent identification information from web page access requests, the network access address is defined as the parsing result of the remote address field representing the access source in the web page access request log record. The remote address field parsing process reads the remote address field from the request header field set and performs network access address format verification. The network access address format verification adopts a combination of character splitting and numerical range verification and outputs the network access address verification result. If the network access address verification result is successful, the remote address field is parsed into a network access address and written into the network access address field. If the network access address verification result is unsuccessful, the network access address field is written with an empty value and written into the network access address exception flag field. The user agent identification information is defined as the original string of the user agent field in the web page access request log record. The user agent field reading process reads the user agent field from the request header field set and performs whitespace character removal and length limit verification. The length limit verification adopts a maximum length threshold and performs truncation rules when the maximum length threshold is exceeded. The truncation rules retain the prefix character sequence of the user agent field string and write it into the user agent identification information field.

[0042] In this implementation method, which associates network access addresses with user agent identification information and writes them into the data records corresponding to the set of users accessing the honeypot, forming a correspondence between network access addresses and user agent identification information, the data records are stored in the form of associated records and include an access request identifier field, a network access address field, a user agent identification information field, a request post sequence number field, a judgment timestamp field, and a hit type field. The hit type field is used to distinguish between hits on invisible honeypot entries and hits on visible honeypot posts, and is determined jointly by the exclusion verification result field in the judgment result record and the honeypot post type field in the honeypot post data table. The association writing process takes the set of webpage access requests hit by the honeypot as input and reads the access request identifier field one by one. The process of reading one by one is carried out in the judgment result. The system performs a reverse lookup based on the access request identifier field in the storage medium to obtain the request post sequence number field and the judgment timestamp field. During the read-by-read process, it retrieves the network access address field and user agent identifier information field from the web page access request log record. After the read-by-read process is completed, it generates an associated record and writes it to the network access address and user agent identifier information mapping storage medium. The network access address and user agent identifier information mapping storage medium adopts a key-value mapping structure, with the network access address field as the key and the user agent identifier information field as the value. During the write phase, the key-value mapping structure allows the same network access address field to correspond to multiple user agent identifier information fields, and stores multiple user agent identifier information fields in a set field to reflect the diversity of access sources.

[0043] The integrity maintenance method for the correspondence between network access addresses and user agent identification information includes write idempotency constraints and missing field handling rules. Write idempotency constraints mean that the associated records of the same access request identification field are only allowed to be written once, and this is achieved through uniqueness verification of the access request identification field. Missing field handling rules mean that when the network access address field or the user agent identification information field is empty, the associated record is written to the missing field marker field, and the key-value mapping is skipped in the network access address and user agent identification information correspondence storage medium to avoid uninterpretable results generated by whitelist matching. The value of the missing field marker field is limited to missing network access address and missing user agent identification information, and it is written together with the access request identification field to ensure audit traceability.

[0044] When referencing the mapping relationship between network access addresses and user agent identification information, the matching input is based on the network access address field and the user agent identification information field. After the honeypot hits the user access set and completes the writing of the mapping relationship between network access addresses and user agent identification information, it retains the access request identification field set and the hit type field set, and shares the access request identification field with the storage medium of the mapping relationship between network access addresses and user agent identification information to form a closed-loop data link. The closed-loop data link consists of the generation of judgment result records, the generation of honeypot hit user access set, the extraction of honeypot hit webpage access request set, the reading of network access address field, the reading of user agent identification information field, and the writing of the mapping relationship between network access addresses and user agent identification information, and ensures that the mapping relationship between network access addresses and user agent identification information can be directly read as a processing object.

[0045] S5: Match the user's user agent identifier information with the preset search engine user agent whitelist to determine whether the user belongs to a non-search engine crawler, and block the network access address of the non-search engine crawler for a preset duration, including: Based on the correspondence between network access addresses and user agent identification information, user agent identification information is extracted and matched with a preset search engine user agent whitelist to generate whitelist matching results. A set of non-search engine crawler web access addresses is formed by filtering web access addresses based on whitelist matching results; The system blocks the network access addresses of non-search engine crawlers for a preset duration and writes the network access address, the start time of the block, and the end conditions of the block into the network access address blocking record.

[0046] Specifically, in the implementation of extracting user agent identification information based on the correspondence between network access addresses and user agent identification information and performing matching with a preset search engine user agent whitelist to generate whitelist matching results, the correspondence between network access addresses and user agent identification information is stored in a key-value mapping structure, with the network access address field as the key and the user agent identification information field as the value. The user agent identification information extraction process reads the network access address field one by one from the key-value mapping structure and reads the user agent identification information field simultaneously. After the user agent identification information field enters the normalization process, it performs whitespace character removal, invisible character removal, and length consistency processing. The length consistency processing adopts a maximum length threshold, and when the maximum length threshold is exceeded, a truncation rule is executed. The truncation rule retains the prefix character sequence of the user agent identification information field and generates normalized user agent identification information. The default search engine user agent whitelist is stored as a set of whitelist entries, including a whitelist entry field and a whitelist category field. The whitelist entry field contains the complete string or string prefix of the search engine user agent identification information. The whitelist category field is used to mark different search engine categories and provide a classification basis for auditing. The default search engine user agent whitelist is generated using a maintenance process that combines manual entry and historical access verification. In the manual entry stage, the trusted search engine user agent identification information string is written and the maintenance timestamp field is recorded. In the historical access verification stage, user agent identification information fields with stable access frequency and trusted sources are selected from existing access logs and manually reviewed before being written into the whitelist entry field to ensure the stability of the whitelist entry set. The whitelist matching process takes standardized user agent identification information as input and performs matching on a preset search engine user agent whitelist. The matching rules include exact matching rules and prefix matching rules. The exact matching rule determines that the standardized user agent identification information is completely consistent with the whitelist entry field. The prefix matching rule determines that the standardized user agent identification information includes the whitelist entry field in a prefix manner. The matching rule selection order follows the principle of prioritizing exact matching rules, and the prefix matching rule is executed when the exact matching rule fails. The matching output generates whitelist matching results and stores them in a structured record. The structured record includes a network access address field, a user agent identification information field, a whitelist matching flag field, a whitelist category field, and a matching timestamp field. The whitelist matching flag field is limited to a successful match or a failed match and must be consistent with the filtering conditions.

[0047] In the implementation of forming a non-search engine crawler web access address set based on whitelist matching results, the whitelist matching results enter the filtering process and are grouped according to the whitelist matching marker field. The filtering process reads the structured records that failed to match the whitelist matching marker field and extracts the web access address field to form a candidate web access address sequence. The candidate web access address sequence enters the deduplication and aggregation process to generate a non-search engine crawler web access address set. The deduplication and aggregation process adopts a set data structure and uses the web access address field as the unique element. The deduplication and aggregation process performs an existence query on each candidate web access address sequence and writes it into the set data structure if it does not exist to avoid duplicate writing. The deduplication and aggregation process simultaneously retains the association index between the web access address field and the user agent identification information field so that the user agent identification information field can be reused when blocking records are written. The filtering process also includes abnormal data filtering rules. These rules are used to process structured records where the network access address field or the user agent identification information field is empty. The abnormal data filtering rules write structured records with empty network access address fields into the abnormal matching record set and skip writing them during the generation stage of the network access address set by non-search engine crawlers. The abnormal data filtering rules write structured records with empty user agent identification information fields into the abnormal matching record set and mark the whitelist matching mark field as a failed match for manual auditing. The abnormal matching record set includes the network access address field, user agent identification information field, abnormal type field, and abnormal timestamp field and is used to maintain the preset search engine user agent whitelist.

[0048] In an implementation method for blocking a set of non-search engine crawler web access addresses for a preset duration and writing the web access address, the start time of the block, and the termination condition of the block into the web access address blocking record, the preconditions for executing the preset duration block include the validity verification of the web access address field within the set of non-search engine crawler web access addresses. The validity verification of the web access address field adopts the web access address format verification and outputs a verification pass flag. When the verification pass flag is set, the blocking rule generation process is initiated. The blocking rule generation process generates a blocking action record for each web access address field and writes it into the blocking queue to be executed. The blocking queue to be executed includes the web access address field, the blocking start time field, the preset duration field, and the blocking termination condition field. The blocking start time field is taken from the server's current time and stored in a unified time format. The preset duration field is set using a rule-based process combining historical blocking effect statistics and access density statistics. Historical blocking effect statistics read historical blocking records from network access address blocking records and calculate the interval between repeated blocking and the frequency of recurrence. Access density statistics read the number of access request identifier fields corresponding to the same network access address field from access logs and calculate the access density distribution within a unit time window. The rule-based process maps historical blocking effect statistics and access density statistics to preset duration fields and forms a configurable rule table. The rule table supports selecting different preset duration fields according to access density levels. Example values ​​for the preset duration field are provided after the rule-based process is completed; for example, it can be set to 24 hours. The blocking termination condition field is set using a combination of time-expiration termination and manual removal termination. Time-expiration termination is triggered when the blocking start time field plus the preset duration field reaches the termination time. Manual removal termination is triggered when maintenance personnel output a removal command on a preset prompt platform and pass command verification. Command verification uses network access address field consistency verification and records the removal timestamp field to ensure traceability.

[0049] After the blocking action record enters the blocking execution process, the blocking execution process writes the network access address field into the access control rule set and sets the blocking start time field and blocking termination condition field. The access control rule set is used to execute the denial policy when a web page access request arrives. The denial policy matches the network access address field in the web page access request log record with the network access address field in the access control rule set, and blocks the access request and writes it to the blocking log record when the match is successful. The blocking log record contains the network access address field, access request identifier field, and blocking timestamp field and is used for auditing. The network access address blocking record writing process is executed after the blocking execution process is completed. The network access address blocking record writing process writes the network access address field, blocking start time field, preset duration field, blocking termination condition field, whitelist matching flag field, and whitelist category field into the network access address blocking record. The whitelist matching flag field is taken from the structured record of the whitelist matching result and retains the value of the match failure. The whitelist category field is written with an empty value when the whitelist matching flag field is a match failure and is written with the corresponding whitelist category field when the whitelist matching flag field is a match success, so as to distinguish wrongful blocking and auditing. The network access address blocking record supports idempotency verification during the maintenance phase. Idempotency verification uses the network access address field and the blocking start time field as a combined unique key and executes an update policy when writing repeatedly. The update policy allows updating the blocking termination condition field and the release timestamp field and prohibits overwriting the existing blocking start time field to maintain the stability of historical links.

[0050] Specifically, a set of user agent identification information is extracted based on the correspondence between network access addresses and user agent identification information; User agent identification information access frequency data is generated by summarizing network access address blocking records based on user agent identification information sets. A high-frequency user agent identification information set is formed by filtering high-frequency user agent identification information based on access frequency data of user agent identification information; The set of high-frequency user agent identification information and the access frequency data of user agent identification information are submitted to the preset prompt platform to generate a queue of manual review tasks.

[0051] Specifically, the implementation process for extracting the user agent identification information set based on the correspondence between network access addresses and user agent identification information is as follows: Access the data structure that stores the correspondence between network access addresses and user agent identification information, and iterate through all the network access addresses stored therein. When iterating through each network access address, extract the user agent identification information associated with the correspondence. Summarize the extracted user agent identification information into a user agent identification information set to generate a user agent identification information set.

[0052] The specific implementation method for generating user agent identification information access frequency data based on the aggregation of network access address blocking records using user agent identification information sets is as follows: The system accesses and extracts the data structure storing network access address blocking records. These records contain information such as the blocked network access address, the start time of the block, and the duration of the block. Each user agent identifier in the user agent identifier information set is processed individually. Using the user agent identifier as a keyword, each network access address blocking record is queried to determine if the user agent identifier corresponding to the blocked network access address matches the keyword. A successful match indicates that the network access address corresponding to the user agent identifier has been blocked, and the number of successful matches is statistically summarized. By processing each user agent identifier in the user agent identifier information set in this way, the system ultimately generates user agent identifier access frequency data.

[0053] User agent identification information access frequency data is implemented in key-value pair format, where the key is the user agent identification information, and the value is the cumulative number of times or frequency index of the network access address associated with the corresponding user agent identification information has been blocked. The access frequency index is calculated by counting the total number of times the network access address associated with the corresponding user agent identification information has been blocked.

[0054] The specific implementation process of filtering high-frequency user agent identification information based on access frequency data to form a high-frequency user agent identification information set is as follows: Analyze the access frequency indicators recorded in the user agent identification information access frequency data to determine the threshold setting method for the user agent identification information access frequency indicators. For example, use historical blocking data to calculate the statistical distribution pattern of the user agent identification information access frequency data, and use percentile analysis method. When the access frequency indicator corresponding to the user agent identification information exceeds the top 10 percentile of the historical data distribution, it is identified as high-frequency user agent identification information.

[0055] Based on the above threshold setting method, the user agent identification information access frequency data is traversed, and user agent identification information that reaches or exceeds the predetermined access frequency index threshold is extracted and summarized into the high-frequency user agent identification information set to generate the high-frequency user agent identification information set.

[0056] The specific implementation method for submitting the high-frequency user agent identification information set and user agent identification information access frequency data to the preset prompt platform to generate a manual review task queue is as follows: Define the data structure for manual review tasks, including task identifier, user agent identifier information, access frequency index, number of times the associated network access address has been blocked, submission timestamp, etc.; iterate through each user agent identifier information in the high-frequency user agent identifier information set, extract the corresponding access frequency index and the number of times the associated network access address has been blocked from the user agent identifier information access frequency data, and populate the data structure of the manual review task to form a single manual review task.

[0057] After a single manual review task is created, it is submitted to the manual review task queue of the preset prompting platform. The task submission method includes, but is not limited to: submitting the data structure of the manual review task to the prompting platform in a standardized data exchange format (such as JSON or XML format) by calling the application interface provided by the platform. The manual review task queue is a first-in-first-out task queue, and the manual reviewers process the tasks in the queue in turn.

[0058] Specifically, network access address prefix information is extracted from network access address blocking records to generate a candidate set of network access address segments; Perform frequency statistics on blocking of candidate network access address ranges to generate network access address range frequency data; A set of high-frequency network access address segments is formed by filtering high-frequency network access address segments based on network access address segment frequency data. Write network access address range blocking rules to the set of high-frequency network access address ranges and associate them with network access address blocking records.

[0059] Specifically, the implementation process of extracting network access address prefix information based on network access address blocking records to generate a candidate set of network access address segments is as follows: The system accesses all blocked network access address data stored in the network access address blocking record. Each network access address is represented in standard Internet Protocol (IP) address format, i.e., the dotted decimal representation of an IPv4 address. Each network access address is parsed individually, and each IPv4 address is segmented into four independent fields. Each field represents the network access address prefix information from left to right. Based on the required blocking granularity, an appropriate length of network access address prefix information is extracted from the segmentation results as a network access address segment. When the blocking granularity requirement is coarse, the first three fields are selected as candidate entries for the network access address segment; when the blocking granularity requirement is fine, all four fields are selected as candidate entries for the network access address segment.

[0060] The specific implementation method for generating network access address range frequency data by performing blocking frequency statistics on the candidate set of network access address ranges is as follows: The system accesses candidate entries for network access address segments within the candidate set. For each candidate entry, it performs a statistical traversal, meaning it checks whether each network access address matches a candidate entry in the network access address blocking record. Matching methods include, but are not limited to, exact prefix matching algorithms, where the prefix character sequence of the network access address string matches the character sequence of the candidate entry. After completing the traversal and matching, it counts the total number of successful matches for each candidate entry, forming network access address segment frequency data.

[0061] Network access address range frequency data is stored in key-value pairs, where the key is the candidate entry for the network access address range and the value is the cumulative number of times the corresponding entry has been successfully blocked in the network access address blocking record.

[0062] The specific implementation process of filtering high-frequency network access address segments based on network access address segment frequency data to form a high-frequency network access address segment set is as follows: Methods for determining the threshold for filtering high-frequency network access address segments include, but are not limited to, percentile methods based on statistical analysis of historical blocking records. For example, the blocking frequency of each network access address segment in the historical blocking record data is sorted, and the blocking frequency of the top 10% of network access address segments is selected as the threshold. For example, if the top 10% of network access address segments in the historical blocking frequency has been blocked 50 times, then 50 times is used as the threshold for high-frequency network access address segments. The network access address segment frequency data is traversed, and for each candidate network access address segment, the cumulative number of blocking times is checked one by one to see if it reaches or exceeds the set threshold. If it does, it is included in the set of high-frequency network access address segments.

[0063] The specific implementation method for writing network access address range blocking rules into a set of high-frequency network access address ranges and associating them with network access address blocking records is as follows: Define the data structure for blocking rules for network access address ranges, including the blocking rule identifier, network access address range, blocking start time, blocking duration, rule effective and end time, and associated network access address blocking record identifier information; create blocking rules for each network access address range within the set of high-frequency network access address ranges, call the server-side network firewall interface, and based on the network access address range, deny or block all network access requests within the corresponding network access address range by configuring blocking rules.

[0064] The duration of a ban is determined based on a comprehensive analysis of the ban frequency and historical ban effectiveness. For example, if historical ban effectiveness analysis shows that a 24-hour ban can achieve the best protection effect, then the ban duration can be set to 24 hours. The ban start time is set to the current server time, which is the actual time when the ban rule is created and activated.

[0065] After the blocking rules are created, the identification information of all original network access address blocking records associated with each blocking rule is explicitly recorded to ensure that there is a correspondence between the blocking rules and historical data.

[0066] During the effective period of the blocking rule, the server-side firewall rules monitor network access requests in real time according to the network access address range specified in the blocking rule data structure, and reject network access requests that fall into the set of high-frequency network access address ranges; when the effective termination time of the blocking rule arrives, the network access address range blocking rule is automatically released, thereby completing the full lifecycle management of the blocking rule.

[0067] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. The semiconductor medium can be a solid-state drive.

[0068] Those skilled in the art will recognize that the modules and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0069] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and modules described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0070] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or modules may be electrical, mechanical, or other forms.

[0071] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0072] In addition, the functional modules in the various embodiments of this application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module.

[0073] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0074] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0075] In conclusion, the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for identifying and protecting web crawlers based on honeypot network security technology, characterized in that, Includes the following steps: S1: Select a predetermined numbered item from the webpage information list, set the CSS style to hidden, generate invisible honeypot items in the form of webpage code, and randomly select some numbers from the consecutive webpage post numbers as visible honeypot posts. S2: When a user accesses a webpage, obtain the post number in the access request link and query the honeypot post data table. By determining whether the post number is in the honeypot post data table, it can be determined whether the user has crawled invisible honeypot entries in the form of webpage code. S3: When a user accesses a webpage, obtain the post number in the access request link and query the honeypot post data table. By determining whether the post number is in the honeypot post data table, it can be determined whether the user has crawled randomly selected visible honeypot posts. S4: For users who have been identified as crawling webpage code-form invisible honeypot entries and randomly selected visible honeypot posts, extract the corresponding network access address and user agent identification information. S5: Match the user agent identification information of the accessing user with the preset search engine user agent whitelist to determine whether the accessing user belongs to a non-search engine crawler, and block the network access address of the non-search engine crawler for a preset duration.

2. The web crawler identification and protection method based on honeypot network security technology according to claim 1, characterized in that, S1, specifically: Extract consecutive webpage post numbers from the webpage information list to form a candidate number set; Randomly select honeypot post serial numbers from the candidate serial number set to form a honeypot post serial number set and write it into the honeypot post data table; Select predetermined numbered entries from the honeypot post sequence number set to form a predetermined sequence numbered entry set; Set the CSS style to hide the web page information list entries corresponding to the predetermined sequence number entry set, and write it into the web page information list web page code to generate invisible honeypot entries in the form of web page code, and set visible honeypot posts according to the honeypot post sequence number set.

3. The web crawler identification and protection method based on honeypot network security technology according to claim 2, characterized in that, S2, specifically: Extract the post sequence number from the webpage access request link initiated by the user to form the request post sequence number; Perform a set ownership check on the request post sequence number and the honeypot post sequence number set in the honeypot post data table to generate an ownership check result; When the attribution verification result indicates that the request post sequence number is located in the honeypot post sequence number set, the invisible honeypot entry in the form of webpage code crawled by the accessing user is determined.

4. The web crawler identification and protection method based on honeypot network security technology according to claim 3, characterized in that, S3, specifically: Extract the post sequence number from the webpage access request link initiated by the user to form the request post sequence number; Perform a set ownership check on the request post sequence number and the honeypot post sequence number set in the honeypot post data table to generate an ownership check result; When the attribution verification result indicates that the requested post sequence number is in the honeypot post sequence number set but not in the predetermined sequence number entry set, it is determined that the accessing user crawled randomly selected visible honeypot posts.

5. The web crawler identification and protection method based on honeypot network security technology according to claim 4, characterized in that, S4, specifically: The results of crawling invisible honeypot entries in the form of web page code by the determined users are combined with the results of crawling randomly selected visible honeypot posts by the determined users to generate a set of honeypot-hit users. Extract webpage access requests corresponding to the honeypot-hit user set based on the honeypot-hit user set; Based on the web page access request, read the network access address and read the user agent identification information; The network access address is associated with the user agent identification information and written into the data record corresponding to the set of users accessing the honeypot, thus forming a correspondence between the network access address and the user agent identification information.

6. The web crawler identification and protection method based on honeypot network security technology according to claim 5, characterized in that, S5, specifically: Based on the correspondence between network access addresses and user agent identification information, user agent identification information is extracted and matched with a preset search engine user agent whitelist to generate whitelist matching results. A set of non-search engine crawler web access addresses is formed by filtering web access addresses based on whitelist matching results; The system blocks the network access addresses of non-search engine crawlers for a preset duration and writes the network access address, the start time of the block, and the end conditions of the block into the network access address blocking record.

7. A method for identifying and protecting web crawlers based on honeypot network security technology according to claim 6, characterized in that, Extract the set of user agent identification information based on the correspondence between network access addresses and user agent identification information; User agent identification information access frequency data is generated by summarizing network access address blocking records based on user agent identification information sets. A high-frequency user agent identification information set is formed by filtering high-frequency user agent identification information based on access frequency data of user agent identification information; The set of high-frequency user agent identification information and the access frequency data of user agent identification information are submitted to the preset prompt platform to generate a queue of manual review tasks.

8. A method for identifying and protecting web crawlers based on honeypot network security technology according to claim 7, characterized in that, Based on network access address blocking records, extract network access address prefix information to generate a candidate set of network access address segments; Perform frequency statistics on blocking of candidate network access address ranges to generate network access address range frequency data; A set of high-frequency network access address segments is formed by filtering high-frequency network access address segments based on network access address segment frequency data. Write network access address range blocking rules to the set of high-frequency network access address ranges and associate them with network access address blocking records.

Images