Malicious web crawler interception method, electronic equipment, storage medium and program product

Abstract

The invention provides a malicious web crawler interception method, electronic equipment, a readable storage medium and a computer program product. The malicious web crawler interception method comprises the following steps: acquiring a malicious web crawler account and / or a service request feature value associated with the malicious web crawler account; receiving a service request, wherein the service request comprises an initiator account and an initiator service request feature value; matching the initiator account with the malicious web crawler account, and / or matching the initiator service request feature value with a service request feature value associated with the malicious web crawler account; and carrying out malicious web crawler interception on the service request under the condition that the initiator account is matched with the malicious web crawler account or the initiator service request feature value is matched with the service request feature value associated with the malicious web crawler account.

Description

Technical Field

[0001] This disclosure relates to a method for intercepting malicious web crawlers, electronic devices, storage media, and program products. Background Technology

[0002] With the rapid development of internet technology, more and more enterprises are gradually transforming towards digitalization and online operations. Along with this digitalization, enterprises are storing and displaying increasingly more data, and people can search for information and data of interest on the internet, in which web crawlers play a significant role.

[0003] However, malicious web crawlers often scrape information indiscriminately, which can easily cause risks such as resource consumption and data leakage to enterprise servers. Summary of the Invention

[0004] This disclosure provides a method for intercepting malicious web crawlers, an electronic device, a storage medium, and a program product.

[0005] According to one aspect of this disclosure, a method for intercepting malicious web crawlers is provided, comprising: obtaining malicious web crawler accounts and / or business request feature values ​​associated with the malicious web crawler accounts, wherein the malicious web crawler accounts are multiple accounts that have a total number of requests for the same business that is greater than or equal to a total request count threshold and are associated with the same business request feature value; receiving a business request, wherein the business request includes an initiating account and an initiating business request feature value; matching the initiating account with the malicious web crawler accounts, and / or matching the initiating business request feature value with the business request feature value associated with the malicious web crawler accounts; and, if the initiating account matches the malicious web crawler accounts, or the initiating business request feature value matches the business request feature value associated with the malicious web crawler accounts, intercepting the malicious web crawler in the business request.

[0006] According to one technical solution, malicious web crawler accounts are pre-identified based on the total number of requests from multiple accounts and the common associated business request characteristics. Then, when a new business request is received, by matching the initiator account in the business request with the malicious web crawler account and / or matching the initiator's business request characteristics with the malicious web crawler's business request characteristics, it is possible to accurately determine whether the initiator is a malicious web crawler. This allows for the timely detection of malicious web crawlers making low-frequency requests based on an account pool, reducing the possibility of misjudging legitimate users. This provides data support for malicious web crawler interception and can promptly prevent malicious web crawlers from consuming enterprise server resources and stealing data.

[0007] According to at least one embodiment of the malicious web crawler interception method of this disclosure, after matching the initiator account with the malicious web crawler account, the method further includes: when the initiator account and the malicious web crawler account do not match, and the initiator's business request feature value does not match the business request feature value associated with the malicious web crawler account, generating a knowledge graph with the initiator account and the initiator's business request feature value as nodes, the relationship between the initiator account and the initiator's business request feature value as edges, and the number of times the initiator account requests the business corresponding to the business request as an attribute, the knowledge graph is used to identify the malicious web crawler account.

[0008] According to the technical solution of this embodiment, when the initiating account and the malicious web crawler account do not match, and the feature value of the initiating business request does not match the feature value of the business request associated with the malicious web crawler account, a knowledge graph is generated. This knowledge graph can structurally record potential malicious web crawler behaviors, thus facilitating the subsequent identification of malicious web crawlers. Compared with analyzing malicious web crawler behavior through logs, the knowledge graph can intuitively express the association between different accounts and the same business request feature value, making it easier to uncover hidden malicious web crawler behaviors. This method improves the adaptive capability of malicious web crawler identification, provides a data foundation for dynamically updating the malicious web crawler account database, and offers an efficient method for behavior analysis in complex network environments.

[0009] According to at least one embodiment of the malicious web crawler interception method of this disclosure, after generating a knowledge graph, the method further includes: determining the total number of requests made by associated accounts to the business based on the initiator's business request feature value, wherein the associated accounts are multiple accounts associated with the initiator's business request feature value; and if the total number of requests is greater than or equal to the total number of requests threshold, the associated accounts are used as new malicious web crawler accounts.

[0010] According to the technical solution of this embodiment, the total number of requests made by associated accounts to the business is determined by knowledge graph. If the total number of requests is greater than or equal to the threshold of the total number of requests, the associated accounts are marked as malicious web crawler accounts. This enables timely detection of malicious web crawlers that make low-frequency requests based on the account pool, providing data support for subsequent interception of malicious web crawlers.

[0011] According to at least one embodiment of the malicious web crawler interception method of this disclosure, determining the total number of requests made by associated accounts to the service from the knowledge graph includes: determining the number of times the service has been requested; and if the number of times the service has been requested is greater than or equal to a threshold number of requests, determining the total number of requests made by associated accounts to the service from the knowledge graph.

[0012] According to the technical solution of this embodiment, the total number of requests for a business is determined from the knowledge graph only when the number of requests for a business is greater than or equal to the threshold number of requests, rather than determining the total number of requests for a business by an associated account in real time. This optimizes the allocation of computing resources, avoids over-analysis of low-risk businesses, and allows for better focus on high-risk businesses, effectively reducing system overhead.

[0013] According to at least one embodiment of the malicious web crawler interception method of this disclosure, before using the associated account as a new malicious web crawler account, the method further includes: determining the number of times the service is requested within a unit time period; if the number of requests within the unit time period is greater than or equal to a set threshold, using a first threshold as the total request count threshold; and if the number of requests within the unit time period is less than the set threshold, using a second threshold as the total request count threshold, wherein the second threshold is less than the first threshold.

[0014] According to the technical solution of this embodiment, by dynamically selecting a first threshold or a second threshold as the total request count threshold based on the number of requests made to the service within a unit of time, adaptive adjustment of the total request count threshold is achieved. Compared with a fixed total request count threshold, this solution can dynamically optimize the malicious web crawler detection standard according to traffic levels. It relaxes the total request count threshold during high traffic to reduce false positives, and tightens it during low traffic to improve sensitivity, thereby enhancing adaptability to different traffic scenarios and effectively balancing detection accuracy and user experience.

[0015] According to at least one embodiment of the malicious web crawler interception method of this disclosure, the malicious web crawler interception of the service request includes: preventing the forwarding of the service request to the service corresponding to the service request; and sending a malicious web crawler interception prompt to the initiator of the service request.

[0016] According to the technical solution of this embodiment, by preventing the forwarding of malicious web crawler service requests, the normal operation of the business system is protected. At the same time, by sending an interception prompt to the initiator, the initiator can be made aware of the progress of the service request.

[0017] According to at least one embodiment of the malicious web crawler interception method of this disclosure, after matching the initiator account with the malicious web crawler account, the method further includes: if the initiator account does not match the malicious web crawler account and the initiator's business request feature value does not match the business request feature value associated with the malicious web crawler account, forwarding the business request to the business corresponding to the business request; and if the response data returned by the business is received, sending the response data to the initiator of the business request.

[0018] According to the technical solution of this embodiment, when the initiator's account does not match the malicious web crawler account, and the initiator's business request feature value does not match the business request feature value associated with the malicious web crawler account, the business request of the initiator is responded to normally, maintaining the integrity of the business process, thereby ensuring that the business requests of normal users can be processed smoothly and guaranteeing the user experience.

[0019] According to another aspect of this disclosure, an electronic device is provided, comprising: a memory storing a computer program; and a processor executing the computer program stored in the memory, such that the processor performs a malicious web crawler interception method according to any embodiment of this disclosure.

[0020] According to another aspect of this disclosure, a readable storage medium is provided, wherein a computer program is stored therein, which, when executed by a processor, is used to implement the malicious web crawler interception method of any embodiment of this disclosure.

[0021] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements a malicious web crawler interception method according to any embodiment of this disclosure. Attached Figure Description

[0022] The accompanying drawings illustrate exemplary embodiments of the present disclosure and, together with the description thereof, serve to explain the principles of the present disclosure. These drawings are included to provide a further understanding of the present disclosure and are incorporated in and constitute a part of this specification.

[0023] Figure 1 This is a flowchart illustrating a method for intercepting malicious web crawlers according to one embodiment of the present disclosure.

[0024] Figure 2 This is a schematic diagram illustrating the process of intercepting malicious web crawlers according to one embodiment of the present disclosure.

[0025] Figure 3 This is a schematic diagram illustrating the process of processing a non-malicious web crawler according to one embodiment of the present disclosure.

[0026] Figure 4 This is a schematic diagram illustrating the process of generating a knowledge graph according to one embodiment of the present disclosure.

[0027] Figure 5 This is a schematic diagram illustrating the process of detecting malicious web crawlers according to one embodiment of the present disclosure.

[0028] Figure 6 This is a visualization example of a knowledge graph according to one embodiment of the present disclosure.

[0029] Figure 7 This is a schematic diagram illustrating the process of determining the total number of requests according to one embodiment of this disclosure.

[0030] Figure 8 This is a schematic diagram illustrating the process of determining a threshold for the total number of requests according to one embodiment of the present disclosure.

[0031] Figure 9 This is an interactive schematic diagram of a malicious web crawler interception system according to one embodiment of the present disclosure.

[0032] Figure 10 This is a schematic structural block diagram of a malicious web crawler interception device according to one embodiment of the present disclosure.

[0033] Figure 11 This is a schematic block diagram of an electronic device according to one embodiment of the present disclosure. Detailed Implementation

[0034] The present disclosure will now be described in further detail with reference to the accompanying drawings and examples. It should be understood that the specific examples described herein are for illustrative purposes only and are not intended to limit the scope of the disclosure. Furthermore, it should be noted that, for ease of description, only the parts relevant to the present disclosure are shown in the accompanying drawings.

[0035] It should be noted that, where there is no conflict, the embodiments and features described in this disclosure can be combined with each other. The technical solutions of this disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0036] Malicious web crawlers exist on the internet. These malicious web crawlers do not follow internet crawler protocols and indiscriminately scrape information. Such behavior puts enormous pressure on enterprise servers, causing lag for normal users or even paralyzing the enterprise server. Furthermore, if these malicious web crawlers are controlled by criminals, they may lead to the leakage of enterprise data and damage the enterprise's interests.

[0037] After careful research, the inventors discovered that as web crawler technology becomes more advanced, most malicious web crawlers no longer use a single account to make high-frequency requests to crawl data. Instead, they employ a more dispersed approach, such as using account pools. Each account makes only a few dozen or fewer requests before switching to another account to continue. Therefore, it is impossible to identify such malicious web crawlers that make low-frequency requests based solely on the number of requests. However, the inventors found that these malicious web crawlers that make low-frequency requests based on account pools share common business request characteristics. For example, all malicious web crawler accounts in the account pool use the same IP address when making business requests, and the total number of requests for the same business by multiple malicious web crawler accounts in the entire account pool is far greater than the number of requests by normal accounts. This provides assistance in identifying malicious web crawlers.

[0038] To this end, the present disclosure proposes the following technical solution, wherein malicious web crawler accounts are identified in advance based on the total number of requests from multiple accounts and the common associated business request feature values. Then, when a new business request is received, by matching the initiator account in the business request with the malicious web crawler account and / or matching the initiator's business request feature value with the malicious web crawler's business request feature value, it is possible to accurately determine whether the initiator is a malicious web crawler, promptly detect malicious web crawlers that make low-frequency requests based on the account pool, and thus provide data support for the interception of malicious web crawlers.

[0039] To facilitate description and make the technical solutions of this disclosure easier to understand, the terminology of this disclosure will be explained before describing the technical solutions of this disclosure.

[0040] A web crawler, also known as a web spider, web robot, or web crawler, is a program or script that automatically retrieves information from the World Wide Web according to certain rules.

[0041] Business request: refers to the operational requirement (such as service or data access requirement) initiated by the client to achieve a specific goal or solve a specific problem.

[0042] Business request characteristic value: refers to the specific data value carried in the business request.

[0043] The malicious web crawler interception method disclosed herein can be used by electronic devices to automatically intercept business requests sent by malicious web crawlers. In this disclosure, electronic devices include, but are not limited to, gateways, routers, switches, firewalls, and proxy servers.

[0044] Figure 1 A schematic diagram illustrating the overall flow of a malicious web crawler interception method according to one embodiment of this disclosure is shown. Figure 1The method M100 shown includes steps S110 to S140. This method can be executed by an electronic device such as a gateway.

[0045] In step S110, malicious web crawler accounts and / or business request feature values ​​associated with malicious web crawler accounts are obtained. Malicious web crawler accounts are multiple accounts that have a total number of requests for the same business that is greater than or equal to the total number of requests threshold and are associated with the same business request feature value.

[0046] Malicious web crawler accounts and the business request characteristics associated with them can be predetermined and stored locally on the electronic device or on other devices that the electronic device can interact with, thus facilitating access by the electronic device.

[0047] Business request characteristics may include one or more of the following: the initiator's IP address, the requested path, the requested domain name, and the user-agent. The business request characteristic value is the specific value of the business request characteristic. Identical business request characteristic values ​​mean that the specific values ​​of the same business request characteristic are identical.

[0048] The total number of requests for the same service can be understood as the sum of the number of requests made by each account within a set time period. The set time period can be configured according to requirements. For example, the set time period can be a time window in the form of hourly, daily, or weekly time windows, or it can be a fixed time period, or a time period since the account was registered.

[0049] The total number of requests threshold can be a fixed value set in advance, or it can be a value that is dynamically adjusted based on set rules or algorithms; there is no limitation here.

[0050] In step S120, a service request is received, which includes the initiator's account and the initiator's service request feature value.

[0051] The initiator's account can be a unique customer identifier for the initiator.

[0052] The characteristics of an initiating business request may include one or more of the following: the initiating party's IP address, the requested path, the requested domain name, and the user-agent. The value of the initiating party's business request characteristic is the specific value of the initiating party's business request characteristic.

[0053] Business requests can be in the form of messages, which contain all the details of the business request, such as the target URL, method, header information, parameters, etc.

[0054] In one example, the request line and headers of an HTTP business request message are shown below: GET / a HTTP / 1.1 Host: abcom X-Real-IP: 10.1.1.1 User-Agent: Mozilla / 5.0 Cookie: user-id=10000000123456 The initiator's account is 10000000123456, the initiator's IP address is 10.1.1.1, the user agent is Mozilla / 5.0, the requested domain name is abcom, and the requested path is / a.

[0055] In step S130, the initiator's account is matched with the malicious web crawler account, and / or the initiator's business request feature value is matched with the business request feature value associated with the malicious web crawler account.

[0056] For example, you can match only the initiator's account with the malicious web crawler account, or you can match only the initiator's business request feature value with the business request feature value associated with the malicious web crawler account, or you can match the initiator's account with the malicious web crawler account and the initiator's business request feature value with the business request feature value associated with the malicious web crawler account, respectively.

[0057] Since both the initiating account and the malicious web crawler account are accounts, and the malicious web crawler account includes multiple accounts, it can be determined whether the initiating account is a malicious web crawler by matching the initiating account with each account in the malicious web crawler account. For example, if any account in the malicious web crawler account is the same as the initiating account, it can be determined that the initiating account matches the malicious web crawler account, that is, the initiating account is a malicious web crawler; if none of the accounts in the malicious web crawler account are different from the initiating account, it can be determined that the initiating account does not match the malicious web crawler account.

[0058] Since both the initiator's business request feature value and the business request feature value associated with the malicious web crawler account are specific values ​​of business request features, it is possible to determine whether the initiator is a malicious web crawler by matching the initiator's business request feature value with the specific values ​​of the corresponding business request features in the business request feature values ​​associated with the malicious web crawler account.

[0059] The types of business request features to be matched can be set according to requirements. For example, matching can be performed based on only one specific value among the initiator's IP address, request path, request domain name, and user agent, or matching can be performed based on multiple specific values ​​among the initiator's IP address, request path, request domain name, and user agent.

[0060] For example, if the initiator's business request feature value has the same value of the same type as the business request feature value associated with the malicious web crawler account, it can be determined that the initiator's business request feature value matches the business request feature value associated with the malicious web crawler account, that is, the initiator is determined to be a malicious web crawler; if the initiator's business request feature value does not have the same value of the same type as the business request feature value associated with the malicious web crawler account, it can be determined that the initiator's business request feature value does not match the business request feature value associated with the malicious web crawler account.

[0061] Furthermore, when determining whether an initiator is a malicious web crawler solely by matching the initiator's account with a malicious web crawler account, if the initiator's account and the malicious web crawler account do not match, the initiator can be temporarily considered not to be a malicious web crawler. Similarly, when determining whether an initiator is a malicious web crawler solely by matching the initiator's business request feature value with the business request feature value associated with a malicious web crawler account, if the initiator's business request feature value and the business request feature value associated with a malicious web crawler account do not match, the initiator can be temporarily considered not to be a malicious web crawler. Finally, when matching both the initiator's account with a malicious web crawler account and the initiator's business request feature value with the business request feature value associated with a malicious web crawler account, if both the initiator's account and the business request feature value do not match, the initiator can be temporarily considered not to be a malicious web crawler.

[0062] In step S140, if the initiator's account matches the malicious web crawler account, or if the initiator's business request feature value matches the business request feature value associated with the malicious web crawler account, the business request is intercepted by the malicious web crawler.

[0063] If any of the account and business request characteristics match, it indicates that the initiator is a malicious web crawler. Therefore, by intercepting malicious web crawlers in business requests, it is possible to prevent malicious web crawlers from occupying enterprise server resources and stealing data in a timely manner.

[0064] The malicious web crawler interception method disclosed herein identifies malicious web crawler accounts in advance based on the total number of requests from multiple accounts and the common associated business request feature values. Then, when a new business request is received, the method matches the initiator account in the business request with the malicious web crawler account and / or matches the initiator's business request feature value with the malicious web crawler's business request feature value. This can accurately determine whether the initiator is a malicious web crawler, promptly detect malicious web crawlers making low-frequency requests based on an account pool, reduce the possibility of misjudging normal users, and thus provide data support for malicious web crawler interception. It can also promptly prevent malicious web crawlers from occupying enterprise server resources and stealing data through interception.

[0065] Regarding step S140, in some embodiments of this disclosure, it may also include, as follows: Figure 2 Steps S141 and S142 are shown.

[0066] In step S141, forwarding the service request to the service corresponding to the service request is prevented.

[0067] By preventing business requests from being forwarded to the corresponding business, malicious web crawlers cannot access real business services and obtain business data, thus preventing them from consuming enterprise server resources and stealing data.

[0068] In step S142, a malicious web crawler interception prompt is sent to the party that initiated the business request.

[0069] The specific content of the malicious web crawler interception prompt can be set according to needs. For example, it can only prompt that the request failed, or it can prompt the reason for the request failure along with the request failure.

[0070] The malicious web crawler interception method described above protects the normal operation of the business system by preventing the forwarding of malicious web crawler business requests, and at the same time, it helps the initiator to understand the progress of the business request by sending an interception prompt.

[0071] Regarding step S140, in other embodiments, besides preventing the forwarding of the business request to the corresponding business, false data can also be returned to the initiator of the business request, thereby deceiving malicious web crawlers and preventing them from evading interception after changing their strategies. The false data can be pre-set fixed data, or it can be data obtained by adjusting the requested real data based on pre-set rules; no limitation is made here.

[0072] In some embodiments of this disclosure, after step S130, the following may also be included: Figure 3 Steps S150 and S160 are shown.

[0073] In step S150, if the initiator's account does not match the malicious web crawler's account, and the initiator's business request feature value does not match the business request feature value associated with the malicious web crawler's account, the business request is forwarded to the business corresponding to the business request.

[0074] If the initiating account does not match the malicious web crawler account, and the characteristic value of the initiating business request does not match the characteristic value of the business request associated with the malicious web crawler account, the initiating account can be temporarily considered not to be a malicious web crawler. Therefore, the business request can be forwarded to the business corresponding to the business request, thereby satisfying the initiating account's business needs.

[0075] In step S160, upon receiving the response data returned by the service, the response data is sent to the party that initiated the service request.

[0076] After receiving a service request, the service can automatically return the corresponding response data. The electronic device can then send the received response data to the party that initiated the service request.

[0077] The malicious web crawler interception method described above responds normally to the initiator's business request when the initiator's account does not match the malicious web crawler's account and the initiator's business request feature value does not match the business request feature value associated with the malicious web crawler's account. This maintains the integrity of the business process, thereby ensuring that the business requests of normal users can be processed smoothly and protecting the user experience.

[0078] In some embodiments of this disclosure, after step S130, the following may also be included: Figure 4 The step S170 shown.

[0079] In step S170, if the initiator account does not match the malicious web crawler account, and the initiator's business request feature value does not match the business request feature value associated with the malicious web crawler account, a knowledge graph is generated with the initiator account and the initiator's business request feature value as nodes, the relationship between the initiator account and the initiator's business request feature value as edges, and the number of times the initiator account requests the business corresponding to the business request as an attribute. The knowledge graph is used to identify malicious web crawler accounts.

[0080] The number of times the initiating account requests the corresponding service can be understood as the number of times the initiating account makes requests for the same service within a set time period. For an explanation of the set time period, please refer to the description above; for the sake of brevity, it will not be repeated here.

[0081] In the absence of a knowledge graph, a knowledge graph can be generated by using the initiator's account and the feature value of the initiator's business request as nodes, the relationship between the initiator's account and the feature value of the initiator's business request as edges, and the number of times the initiator's account requests the corresponding business as an attribute. When other business requests are received subsequently, the relevant initiator's account and the feature value of the initiator's business request can be added to this knowledge graph, thereby incrementally updating the knowledge graph. This facilitates the overall analysis of multiple accounts and accurately identifies malicious web crawlers.

[0082] If a knowledge graph already exists, first determine if nodes for the initiator's account or the initiator's business request feature value already exist in the knowledge graph. If no such nodes exist, add the corresponding nodes, edges, and attributes to the knowledge graph. If a node for the initiator's account exists but no node for the initiator's business request feature value exists, add the corresponding node to the knowledge graph and connect it to the node for the initiator's account via an edge, while accumulating the number of times the initiator's account requests the corresponding business request. If a node for the initiator's business request feature value exists but no initiator's account exists, add the corresponding node to the knowledge graph and connect it to the node for the initiator's business request feature value via an edge, while accumulating the number of times the initiator's account requests the corresponding business request. If both an initiator's account and the initiator's business request feature value exist, accumulate the number of times the initiator's account requests the corresponding business request to the knowledge graph.

[0083] In one example, the initiator account is 10001, and the initiator's business request characteristic value is the initiator's IP address 10.1.1.1. This can generate a node corresponding to 10001 and a node corresponding to 10.1.1.1. These two nodes are connected by an edge, which indicates that the IP address logged in by account 10001 is 10.1.1.1.

[0084] In another example, the initiator account is 10002, and the initiator's business request characteristic value is the initiator's IP address 10.2.2.2. This can generate a node corresponding to 10002 and a node corresponding to 10.2.2.2. These two nodes are connected by an edge, which indicates that the IP address logged in by account 10002 is 10.2.2.2.

[0085] If the initiator's account matches a malicious web crawler account, or if the initiator's business request feature value matches a business request feature value associated with a malicious web crawler account, a knowledge graph may not need to be generated, thereby saving resources and avoiding unnecessary expenses.

[0086] The malicious web crawler interception method described above generates a knowledge graph when the initiating account and the malicious web crawler account do not match, and the feature value of the initiating account's business request does not match the feature value of the business request associated with the malicious web crawler account. This knowledge graph can structurally record potential malicious web crawler behaviors, facilitating subsequent identification of malicious web crawlers. Compared to analyzing malicious web crawler behavior through logs, the knowledge graph can intuitively express the association between different accounts and the same business request feature value, making it easier to uncover hidden malicious web crawler behaviors. This method improves the adaptive capability of malicious web crawler identification, provides a data foundation for dynamically updating the malicious web crawler account database, and offers an efficient method for behavior analysis in complex network environments.

[0087] In some embodiments of this disclosure, after step S130, the following may also be included: Figure 5 Steps S180 and S190 are shown.

[0088] In step S180, based on the initiator's business request feature value, the total number of requests made by associated accounts to the business is determined from the knowledge graph. The associated accounts are multiple accounts associated with the initiator's business request feature value.

[0089] The total number of requests made by associated accounts to the same business can be understood as the sum of the number of requests made by each account within a set time period to the same business, among multiple accounts that share the same initiator's business request characteristics. The explanation of the set time period can be found in the description above; for the sake of brevity, it will not be repeated here.

[0090] The number of requests made by each account for the same business within a set time period can be obtained from the attributes of the corresponding node in the knowledge graph.

[0091] In step S190, if the total number of requests is greater than or equal to the total number of requests threshold, the associated account is identified as a new malicious web crawler account.

[0092] Adding a linked account as a new malicious web crawler account can be understood as adding the linked account to an existing list, collection, or database of malicious web crawler accounts. Adding a linked account as a new malicious web crawler account will not affect the existing malicious web crawler accounts.

[0093] If the total number of requests is less than the total number of requests threshold, the total number of requests for the associated account can continue to be monitored.

[0094] In one example, a knowledge graph is like Figure 6As shown, accounts 1001, 1002, and 1003 are associated with IP addresses 8.8.2.1 and 8.1.3.1. Therefore, the total number of requests made by accounts 1001, 1002, and 1003 for the same business will be calculated. If the total number of requests is greater than or equal to the threshold, it means that accounts 1001, 1002, and 1003 are a web crawler group using an account pool. Consequently, accounts 1001, 1002, and 1003 will be subject to malicious web crawling interception.

[0095] In one example, a web crawler group wanted to crawl pages from http: / / abcom / 1.html to http: / / abcom / 10000.html, which would require a total of 10,000 requests. However, the abcom domain has anti-crawling measures; a single account will be banned if it makes more than 500 requests. This means that if the crawler group used a single account to crawl information all at once, it would be quickly banned. Therefore, the group registered 100 accounts and 5 IP addresses: 10.1.1.1, 10.1.1.2, 10.1.1.3, 10.1.1.4, and 10.1.1.5. By making low-frequency requests through these 100 accounts, each account only needed to make 100 requests to achieve the goal of crawling the entire site. During the operation of the malicious web crawler interception method disclosed herein, business requests are monitored in real time, and the initiating account and the initiating business request feature value are linked together in the form of a knowledge graph. By analyzing the knowledge graph, it can be quickly discovered that some of the 100 accounts have the same IP address. Then, if the total number of requests made by the accounts with the same IP address to abcom is greater than or equal to 500, these accounts with the same IP address can be marked as malicious web crawler accounts. Subsequently, any business requests sent by these accounts or business requests carrying these IP addresses will be blocked because they are considered malicious web crawlers, even if the account only makes one request.

[0096] The malicious web crawler interception method described above determines the total number of requests made by associated accounts to the business through a knowledge graph. When the total number of requests is greater than or equal to the threshold of the total number of requests, the associated accounts are marked as malicious web crawler accounts. This enables timely detection of malicious web crawlers that make low-frequency requests based on the account pool, providing data support for subsequent interception of malicious web crawlers.

[0097] Regarding step S180, in some embodiments of this disclosure, it may include, for example... Figure 7 Steps S181 and S182 are shown.

[0098] In step S181, the number of times the service is requested is determined.

[0099] As one possible implementation, the number of requests for a service can be understood as the total number of times the current service is requested by all accounts within a target time period. The target time period can be set according to requirements; for example, the target time period can be a time window in the form of an hour, day, or week, or it can be a fixed time period.

[0100] As another possible implementation, the number of requests for a service can be understood as the total number of times the current service is requested by all accounts within the target period. The target period can be associated with a threshold for the number of requests. For example, every 1000 (i.e., the threshold for the number of requests) requests can be set as the target period. Then, whenever the number of requests for a service reaches 1000, the number of requests for the service is re-accumulated from 0. At the same time, it can trigger the identification of whether the associated accounts in the knowledge graph are malicious web crawler accounts.

[0101] In step S182, if the number of requests for a service is greater than or equal to the request count threshold, the total number of requests made by the associated account for the service is determined from the knowledge graph.

[0102] If the number of requests to a service is less than the request count threshold, monitoring of the number of requests to that service will continue. The request count threshold can be set according to requirements and is not limited here. The request count threshold can be greater than the total request count threshold.

[0103] The malicious web crawler interception method described above determines the total number of requests made by associated accounts to a business from the knowledge graph only when the number of requests to a business is greater than or equal to the threshold number of requests, rather than determining the total number of requests made by associated accounts to a business in real time. This optimizes the allocation of computing resources, avoids over-analysis of low-risk businesses, and allows for better focus on high-risk businesses, effectively reducing system overhead.

[0104] In some embodiments of this disclosure, prior to step S170, the following may also be included: Figure 8 Steps S210 to S230 are shown.

[0105] In step S210, the number of times the service is requested within a unit of time is determined.

[0106] The unit of time can be set according to needs, such as one minute, one hour, one day, one week, etc.

[0107] The number of times a service is requested within a unit of time, that is, the total number of times the current service is requested by all accounts within a unit of time.

[0108] The more requests a service receives per unit of time, the higher its traffic; conversely, the fewer requests a service receives per unit of time, the lower its traffic.

[0109] In step S220, if the number of requests per unit time is greater than or equal to a set threshold, the first threshold is used as the total number of requests threshold.

[0110] In step S230, if the number of requests per unit time is less than a set threshold, the second threshold is used as the total number of requests threshold, and the second threshold is less than the first threshold.

[0111] The threshold, first threshold, and second threshold can be set according to requirements and are not limited here.

[0112] The malicious web crawler interception method described above achieves adaptive adjustment of the total request count threshold by dynamically selecting either a first threshold or a second threshold based on the number of requests made to the service per unit time. Compared to a fixed total request count threshold, this solution can dynamically optimize the malicious web crawler detection standard according to traffic levels. It relaxes the total request count threshold during high traffic to reduce false positives and tightens it during low traffic to improve sensitivity, thereby enhancing adaptability to different traffic scenarios and effectively balancing detection accuracy and user experience.

[0113] It should be noted that the specific values ​​mentioned above are only for illustrating the implementation of this disclosure in detail, and should not be construed as limiting the disclosure. In other examples, implementation methods, or embodiments, other values ​​may be selected based on this disclosure, and no specific limitations are made here.

[0114] Please combine Figure 9 In one example, a malicious web crawler interception system includes a client, a gateway, a crawler analysis engine, a knowledge graph system, a graph database, and a business system.

[0115] In step S301, the client sends the service request to the gateway.

[0116] In step S302, the gateway forwards the service request to the crawler analysis engine.

[0117] In step S303, the crawler analysis engine parses the business request and obtains the initiator's account and the initiator's business request feature value from the business request.

[0118] In step S304, the crawler analysis engine sends the initiator's account and the initiator's business request feature values ​​to the knowledge graph system.

[0119] In step S305, the knowledge graph system checks the graph database to determine whether the initiator is a malicious web crawler. Specifically, the knowledge graph system matches the initiator's account with malicious web crawler accounts, and / or matches the initiator's business request feature values ​​with the business request feature values ​​associated with the malicious web crawler accounts. Based on the matching results, it determines whether the initiator is a malicious web crawler. The specific determination process can be referred to the description above, and will not be repeated here for the sake of brevity. In addition, the knowledge graph system can generate a knowledge graph based on the business request when the initiator is not a malicious web crawler, and store the generated knowledge graph in the graph database. Furthermore, the knowledge graph system can also analyze whether there is a malicious web crawler in the knowledge graph stored in the graph database when the number of requests for the business is greater than or equal to the request count threshold.

[0120] In step S306, the knowledge graph system sends the judgment result to the crawler analysis engine.

[0121] In step S307, the crawler analysis engine determines the detection result based on the judgment result. Specifically, if the initiator is a malicious web crawler, the detection result is blocking; if the initiator is not a malicious web crawler, the detection result is allowing.

[0122] In step S308, the crawler analysis engine sends the detection results to the gateway.

[0123] In step S309, if the detection result is clear, the gateway forwards the service request to the service system.

[0124] In step S310, the business system sends the response data of the business request to the gateway.

[0125] In step S311, if the detection result is "allow", the gateway sends response data to the client.

[0126] In step S312, if the detection result is interception, the gateway prevents the forwarding of the service request to the service system.

[0127] In step S313, if the detection result is blocked, the gateway sends a malicious web crawler interception prompt to the client.

[0128] Based on any of the above embodiments, this disclosure also provides a malicious web crawler interception device.

[0129] Figure 10 This is a schematic block diagram of a malicious web crawler interception device according to one embodiment of the present disclosure.

[0130] like Figure 10As shown, the malicious web crawler interception device includes: an acquisition module 110, used to acquire malicious web crawler accounts and / or business request feature values ​​associated with malicious web crawler accounts, wherein a malicious web crawler account is multiple accounts that have a total number of requests for the same business that is greater than or equal to a total request count threshold and are associated with the same business request feature value; a receiving module 120, used to receive business requests, wherein the business requests include an initiator account and an initiator business request feature value; a matching module 130, used to match the initiator account with the malicious web crawler account, and / or match the initiator business request feature value with the business request feature value associated with the malicious web crawler account; and an interception module 140, used to intercept the business request for malicious web crawling when the initiator account matches the malicious web crawler account, or when the initiator business request feature value matches the business request feature value associated with the malicious web crawler account.

[0131] The aforementioned malicious web crawler interception device can be in the form of computer software, and each module of the aforementioned malicious web crawler interception device can be implemented through computer software modules.

[0132] In some embodiments of this disclosure, the interception module 140 is used to: prevent the forwarding of a business request to the business corresponding to the business request; and send a malicious web crawler interception prompt to the initiator of the business request.

[0133] In some embodiments of this disclosure, the malicious web crawler interception device may further include: a generation module, configured to generate a knowledge graph when the initiator account and the malicious web crawler account do not match, and the initiator's business request feature value does not match the business request feature value associated with the malicious web crawler account, using the initiator account and the initiator's business request feature value as nodes, the relationship between the initiator account and the initiator's business request feature value as edges, and the number of times the initiator account requests the business corresponding to the business as an attribute, the knowledge graph is used to identify the malicious web crawler account.

[0134] In some embodiments of this disclosure, the malicious web crawler interception device may further include: a first determining module, configured to determine the total number of requests made by associated accounts to the business based on the initiator's business request feature value from a knowledge graph, wherein the associated accounts are multiple accounts associated with the initiator's business request feature value; and a marking module, configured to mark the associated accounts as new malicious web crawler accounts when the total number of requests is greater than or equal to a total request number threshold.

[0135] In some embodiments of this disclosure, the first determining module is used to: determine the number of times the service is requested; and if the number of times the service is requested is greater than or equal to a threshold number of requests, determine the total number of requests made by the associated account to the service from the knowledge graph.

[0136] In some embodiments of this disclosure, the malicious web crawler interception device may further include: a second determining module, configured to determine the number of times a service is requested within a unit time period; a third determining module, configured to use a first threshold as a total request count threshold if the number of requests within a unit time period is greater than or equal to a set threshold; and a fourth determining module, configured to use a second threshold as a total request count threshold if the number of requests within a unit time period is less than a set threshold, wherein the second threshold is less than the first threshold.

[0137] In some embodiments of this disclosure, the malicious web crawler interception device may further include: a forwarding module, configured to forward the business request to the business corresponding to the business request when the initiator account does not match the malicious web crawler account and the initiator business request feature value does not match the business request feature value associated with the malicious web crawler account; and a sending module, configured to send the response data to the initiator of the business request when the response data returned by the business is received.

[0138] The specific implementation process of the functions and roles of each module in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0139] The execution entity of the malicious web crawler interception method in the specific embodiments of this disclosure can be an electronic device such as a gateway.

[0140] Therefore, based on any of the above embodiments, this disclosure also provides an electronic device that can execute the malicious web crawler interception method of any of the embodiments described above.

[0141] Figure 11 This is a schematic block diagram of an electronic device 1000 according to one embodiment of the present disclosure.

[0142] The hardware structure of electronic device 1000 can be implemented using a bus architecture. The bus architecture can include any number of interconnect buses and bridges, depending on the specific application and overall design constraints of the hardware. Bus 1100 connects various circuits including one or more processors 1200, memory 1300, and / or hardware modules. Bus 1100 can also connect various other circuits 1400 such as peripherals, voltage regulators, power management circuits, external antennas, etc. Bus 1100 can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Component (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, only one connection line is used in this figure, but this does not indicate that there is only one bus or one type of bus.

[0143] For ease of explanation, certain steps of the above method are described in relation to modules. It should be understood that the corresponding module performing one or more steps of the above method may be one or more hardware modules specifically configured to perform the corresponding step, or implemented by a processor configured to perform the corresponding step, or stored in a computer-readable medium for implementation by a processor, or implemented by some combination thereof.

[0144] This disclosure also provides a readable storage medium storing a computer program that, when executed by a processor, is used to implement the methods described above. A "readable storage medium" can be any means capable of containing, storing, communicating, propagating, or transmitting a program for use by or in conjunction with an instruction execution system, apparatus, or device. More specific examples of a readable storage medium include: an electrical connection with one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and programmable read-only memory (EPROM or flash memory), fiber optic devices, and portable read-only memory (CDROM), etc.

[0145] This disclosure also provides a computer program product, the methods of which can be implemented wholly or partially through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented wholly or partially as a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instructions are loaded and executed, all or part of the processes or functions of this disclosure are performed.

[0146] Computer programs or instructions can be stored in a readable storage medium or transferred from one readable storage medium to another. For example, the computer program or instructions can be transferred from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless means. The readable storage medium can be any available medium capable of access, or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium, such as a floppy disk, hard disk, or magnetic tape; an optical medium, such as a digital video optical disc; or a semiconductor medium, such as a solid-state drive. The computer-readable storage medium can be a volatile or non-volatile storage medium, or it can include both volatile and non-volatile types of storage media.

[0147] Those skilled in the art will understand that embodiments of this disclosure can be provided as methods, apparatus, or computer program products. Therefore, this disclosure can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this disclosure can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0148] This disclosure is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus, and computer program products according to this disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0149] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0150] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0151] In the description of this specification, the references to terms such as "one embodiment / mode," "some embodiments / modes," "example," "specific example," or "some examples," etc., refer to specific features, structures, or characteristics described in connection with that embodiment / mode or example, which are included in at least one embodiment / mode or example of this disclosure. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment / mode or example. Moreover, the specific features, structures, or characteristics described may be combined in any suitable manner in one or more embodiments / modes or examples. Furthermore, without contradiction, those skilled in the art can combine and integrate the different embodiments / modes or examples described in this specification, as well as the features of different embodiments / modes or examples.

[0152] Those skilled in the art should understand that the above embodiments are merely for illustrating the present disclosure and are not intended to limit the scope of the disclosure. Those skilled in the art can make other changes or modifications based on the above disclosure, and these changes or modifications still fall within the scope of the present disclosure.

Claims

1. A method for intercepting malicious web crawlers, characterized in that, include: Obtain malicious web crawler accounts and / or business request feature values ​​associated with the malicious web crawler accounts, wherein the malicious web crawler accounts are multiple accounts that have a total number of requests for the same business that is greater than or equal to a total request count threshold and are associated with the same business request feature values. Receive a service request, wherein the service request includes the initiator's account and the initiator's service request feature value; Match the initiator account with the malicious web crawler account, and / or match the initiator's business request feature value with the business request feature value associated with the malicious web crawler account; as well as If the initiator's account matches the malicious web crawler account, or if the initiator's business request feature value matches the business request feature value associated with the malicious web crawler account, the business request will be intercepted by the malicious web crawler.

2. The method for intercepting malicious web crawlers according to claim 1, characterized in that, After matching the initiator account with the malicious web crawler account, the process also includes: If the initiator account does not match the malicious web crawler account, and the initiator's business request feature value does not match the business request feature value associated with the malicious web crawler account, a knowledge graph is generated using the initiator account and the initiator's business request feature value as nodes, the relationship between the initiator account and the initiator's business request feature value as edges, and the number of times the initiator account requests the business corresponding to the business request as an attribute. The knowledge graph is used to identify the malicious web crawler account.

3. The malicious web crawler interception method according to claim 2, characterized in that, After generating the knowledge graph, the following is also included: Based on the initiator's business request feature value, the total number of requests made by associated accounts to the business is determined from the knowledge graph; the associated accounts are multiple accounts associated with the initiator's business request feature value; and If the total number of requests is greater than or equal to the threshold for the total number of requests, the associated account will be considered a new malicious web crawler account.

4. The malicious web crawler interception method according to claim 3, characterized in that, The total number of requests made by the associated accounts for the business is determined from the knowledge graph, including: Determine the number of times the service has been requested; and If the number of requests for the service is greater than or equal to the request count threshold, the total number of requests made by the associated account for the service is determined from the knowledge graph.

5. The malicious web crawler interception method according to claim 3, characterized in that, Before using the associated account as a new malicious web crawler account, the following is also included: Determine the number of times the service is requested per unit of time; If the number of requests per unit time is greater than or equal to a set threshold, the first threshold is used as the total number of requests threshold; and If the number of requests per unit time is less than the set threshold, the second threshold is used as the total number of requests threshold, and the second threshold is less than the first threshold.

6. The method for intercepting malicious web crawlers according to claim 1, characterized in that, The malicious web crawler interception of the aforementioned service requests includes: Prevent the forwarding of the service request to the service corresponding to the service request; and Send a malicious web crawler interception alert to the party that initiated the business request.

7. The method for intercepting malicious web crawlers according to claim 1, characterized in that, After matching the initiator account with the malicious web crawler account, the process also includes: If the initiating account does not match the malicious web crawler account, and the initiating business request feature value does not match the business request feature value associated with the malicious web crawler account, the business request will be forwarded to the business corresponding to the business request; and Upon receiving the response data returned by the service, the response data is sent to the party that initiated the service request.

8. An electronic device, characterized in that, include: Memory, the memory storing computer programs; as well as A processor that executes a computer program stored in the memory, causing the processor to perform the malicious web crawler interception method according to any one of claims 1 to 7.

9. A readable storage medium, characterized in that, The readable storage medium stores a computer program, which, when executed by a processor, is used to implement the malicious web crawler interception method according to any one of claims 1 to 7.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the malicious web crawler interception method according to any one of claims 1 to 7.

Images