Anti-crawler method based on energy management and related equipment

Abstract

The invention discloses an anti-crawler method based on energy management and related equipment, and belongs to the technical field of communication.The method comprises the steps that when a target request is received, whether the login environment of the target request passes verification or not is judged; if yes, obtaining identification information of the target request, and judging whether the identification information of the target request is in a blacklist; if yes, refusing to respond to the target request; if not, judging whether the identification information of the target request is in a blacklist cache list or not; wherein the blacklist and the blacklist cache list are lists created based on a machine learning algorithm and according to historical logs; identification information of illegal users is stored in the blacklist, and identification information of abnormal users is stored in the blacklist cache list; the historical log is a log corresponding to a historical request for accessing the gateway; and if yes, responding to the target request, and prompting abnormal information. Through the method, crawler behaviors on the network can be effectively limited, and the security of the network is ensured.

Description

Technical Field

[0001] The present invention relates to the field of communication technology, and in particular to an anti-crawler method, device, equipment, program product and medium based on energy management. Background Art

[0002] A crawler is an automated software program whose main task is to browse web pages on the Internet and collect information. With the rapid development of Internet technology, the Internet carries a large amount of valuable information. Crawlers can meet people's demand for information data by crawling information from massive data. For example: information providers crawl price information from online trading websites to provide people with bidding comparison data, and public opinion control departments crawl public speech information from social networks to supervise people's online speech environment.

[0003] Although crawler technology plays an important role in data collection and information acquisition, crawlers have some potential hazards, such as: large-scale crawler activities may cause server overload and reduce website performance; malicious crawlers may occupy bandwidth through a large number of requests, thereby slowing down the access speed of normal users; crawlers can also access and collect personal data on websites, thereby posing the risk of leaking user privacy; crawlers can also use the captured website content for unauthorized network infringement and lead to improper and unfair competition. However, a method for effectively restricting crawler behavior has not yet been proposed, and this problem urgently needs to be solved by those skilled in the art. Summary of the invention

[0004] In view of this, the purpose of the present invention is to provide an anti-crawler method, device, equipment, program product and medium based on energy management to solve the technical problem that the existing technology cannot effectively restrict crawler behavior. The specific scheme is as follows:

[0005] In order to solve the above technical problems, the present invention provides an anti-crawler method based on energy management, which is applied to a gateway and includes:

[0006] When receiving a target request, determining whether the login environment of the target request passes the verification;

[0007] If yes, obtaining the identification information of the target request, and determining whether the identification information of the target request is in the blacklist;

[0008] If so, refusing to respond to the target request;

[0009] If not, determine whether the identification information of the target request is in the blacklist cache list; wherein the blacklist list and the blacklist cache list are lists created based on a machine learning algorithm and according to historical logs; the blacklist list stores the identification information of illegal users, and the blacklist cache list stores the identification information of abnormal users; the historical log is a log corresponding to historical requests for accessing the gateway;

[0010] If yes, respond to the target request and prompt abnormal information.

[0011] Preferably, the process of creating the blacklist list and the blacklist cache list includes:

[0012] Perform crawler detection on the historical log to obtain target detection data;

[0013] Machine learning is performed on the target detection data to obtain target learning data, and the blacklist list and the blacklist cache list are created according to the target learning data.

[0014] Preferably, performing crawler detection on the historical log to obtain target detection data includes:

[0015] The target detection data is obtained by detecting the request frequency of the same user per unit time, the request time interval of the same user per unit time, the ratio of user agents being preset user agents, the ratio of requests carrying cookies, and the ratio of accessing static resources in the historical log.

[0016] Preferably, performing machine learning on the target detection data to obtain target learning data includes:

[0017] The target detection data is subjected to machine learning using a decision tree algorithm to obtain the target learning data.

[0018] Preferably, when receiving the target request, verifying the login environment of the target request includes:

[0019] Triggering the login page to perform a human-machine verification on the login environment of the target request, and determining whether the login environment of the target request passes the verification;

[0020] If yes, feeding back the environment verification code to the login page, so that the target request carries the environment verification code to log in to the gateway;

[0021] When receiving the target request carrying the environment verification code, verifying the environment verification code and determining whether the environment verification code passes the verification;

[0022] If so, continue to execute the step of obtaining the identification information of the target request and determining whether the identification information of the target request is in the blacklist.

[0023] Preferably, after the triggering login page performs a human-machine verification on the login environment of the target request and determines whether the login environment of the target request passes the verification, it further includes:

[0024] If not, feeding back a behavior verification code to the login page, so that the login page uses the behavior verification code to verify the login environment of the target request, and determines whether the login environment of the target request passes the verification of the behavior verification code;

[0025] If yes, continue to execute the step of feeding back the environment verification code to the login page so that the target request carries the environment verification code to log in to the gateway;

[0026] If not, continue to execute the step of feeding back the behavior verification code to the login page, so that the login page uses the behavior verification code to verify the login environment of the target request, and determine whether the login environment of the target request passes the verification of the behavior verification code.

[0027] Preferably, after determining whether the identification information of the target request is in the blacklist cache list, the method further includes:

[0028] The blacklist cache list is updated according to the identification information of the target request.

[0029] In order to solve the above technical problems, the present invention provides an anti-crawler device based on energy management, which is applied to a gateway and includes:

[0030] An environment verification module, used for verifying the login environment of the target request when receiving the target request;

[0031] An identification acquisition module, used for acquiring identification information of the target request if the login environment of the target request passes the verification, and determining whether the identification information of the target request is in the blacklist;

[0032] A request response module, configured to refuse to respond to the target request when the determination result of the identification acquisition module is yes;

[0033] A list judgment module, used for judging whether the identification information of the target request is in a blacklist cache list when the judgment result of the identification acquisition module is no; wherein the blacklist list and the blacklist cache list are lists created based on a machine learning algorithm and according to historical logs; the blacklist list stores the identification information of illegal users, and the blacklist cache list stores the identification information of abnormal users; the historical log is a log corresponding to historical requests for accessing the gateway;

[0034] The information prompt module is used to respond to the target request and prompt abnormal information when the determination result of the list determination module is yes.

[0035] In order to solve the above technical problem, the present invention provides an electronic device, comprising:

[0036] Memory for storing computer programs;

[0037] A processor is used to implement the steps of an anti-crawler method based on energy management as disclosed above when executing the computer program.

[0038] In order to solve the above technical problems, the present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of an anti-crawler method based on energy management as disclosed above.

[0039] In order to solve the above technical problems, the present invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of an anti-crawler method based on energy management as disclosed above are implemented.

[0040] Beneficial effect: In the anti-crawler method provided by the present invention, it is necessary to obtain the log corresponding to the historical request to access the gateway in advance to obtain the historical log. When the historical log is obtained, a blacklist list and a blacklist cache list are created based on the machine learning algorithm and the historical log. Among them, the blacklist list stores the identification information of illegal users, and the blacklist cache list stores the identification information of abnormal users. When the gateway receives the target request, it first verifies the login environment of the target request; if the login environment of the target request passes the verification, the identification information of the target request is obtained, and it is determined whether the identification information of the target request is in the blacklist. If the identification information of the target request is in the blacklist, it means that the target request is a malicious crawler, and the gateway will refuse to respond to the target request. If the identification information of the target request is not in the blacklist, the gateway will continue to determine whether the identification information of the target request is in the blacklist cache. If the identification information of the target request is in the blacklist cache, the gateway will prompt abnormal information.

[0041] Through the method provided by the present application, because the blacklist list and the blacklist cache list are created based on the log corresponding to the historical request of the gateway, and the blacklist list and the blacklist cache list are used to constrain and limit the request to access the gateway, the method can effectively limit the crawler behavior on the network and ensure the security of the network. In addition, the method is based on the asynchronous analysis of the historical log when analyzing the crawler behavior on the network, so that it will not affect the performance of normal users when accessing the network. At the same time, through the machine learning algorithm, the historical log is deeply analyzed, and the legitimacy of the historical request to access the gateway can be more accurately identified and analyzed, thereby ensuring the accuracy and reliability of the creation results of the blacklist list and the blacklist cache list, so that the accuracy of the identification of the crawler can be further improved. In addition, by designing the blacklist cache list, a certain cache mechanism is provided for possible misjudgment, which can not only reduce the impact of abnormal situations on the normal user access process, but also reduce the impact and intervention of human factors on the creation results of the blacklist cache list, thereby further reducing the probability of misjudgment of network crawlers.

[0042] Correspondingly, the anti-crawler device, equipment, program product and medium based on energy management provided by the present invention also have the above-mentioned beneficial effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0044] Figure 1 A flowchart of an anti-crawler method based on energy management provided by an embodiment of the present invention;

[0045] Figure 2 This is a schematic diagram of creating a blacklist list and a blacklist cache list based on the historical logs of the gateway;

[0046] Figure 3 A schematic diagram of asynchronously storing the historical logs of the gateway;

[0047] Figure 4 A schematic diagram of content evaluation for crawler detection of historical logs of a gateway;

[0048] Figure 5 This is a schematic diagram of the principle of the decision tree model;

[0049] Figure 6 A flowchart of another anti-crawler method based on energy management provided by an embodiment of the present invention;

[0050] Figure 7 Schematic diagram of three types of behavior verification codes provided by an embodiment of the present invention;

[0051] Figure 8 A structural diagram of an anti-crawler device based on energy management provided by an embodiment of the present invention;

[0052] Fig. 9 A structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0053] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0054] See also Figure 1 , Figure 1 A flowchart of an anti-crawler method based on energy management provided by an embodiment of the present invention, the method comprising:

[0055] Step S11: When a target request is received, the login environment of the target request is verified;

[0056] Step S12: If the login environment of the target request passes the verification, the identification information of the target request is obtained, and it is determined whether the identification information of the target request is in the blacklist; if so, step S13 is executed; if not, step S14 is executed;

[0057] Step S13: Refuse to respond to the target request;

[0058] Step S14: determine whether the identification information of the target request is in the blacklist cache list; if so, execute step S15;

[0059] Among them, the blacklist list and the blacklist cache list are lists created based on machine learning algorithms and according to historical logs; the blacklist list stores the identification information of illegal users, and the blacklist cache list stores the identification information of abnormal users; the historical log is the log corresponding to the historical request to access the gateway;

[0060] Step S15: respond to the target request and prompt abnormal information.

[0061] In this embodiment, an anti-crawler method is provided, by which the crawler behavior on the network can be effectively restricted and the network security can be ensured. The method is described with the gateway as the execution subject.

[0062] In this method, it is necessary to obtain the logs corresponding to the historical requests for accessing the gateway in advance to obtain the historical logs. After obtaining the historical logs of the gateway, a blacklist list and a blacklist cache list are created based on the machine learning algorithm and the historical logs of the gateway. The blacklist list stores the identification information (user name and IP address) of illegal users, and the blacklist cache list stores the identification information of abnormal users.

[0063] Since the history log of the gateway records various events during the operation of the gateway, by analyzing the history log of the gateway, various user behavior patterns can be determined and potential security threats to the gateway can be found. When creating a blacklist and a blacklist cache list, it is equivalent to crawling the history log of the gateway and finding illegal and abnormal users who access the gateway based on the crawler detection results.

[0064] In other words, the blacklist is a permanent blocking mechanism. The blacklist is used to prevent known malicious users or IP addresses from accessing the gateway. The blacklist contains the usernames and IP addresses that have been confirmed as malicious users. The users or IP addresses in the blacklist have made malicious attacks on the gateway in the past. Once a user or IP address is added to the blacklist, these users will not be able to access the gateway again unless the administrator manually unblocks them. Obviously, creating a blacklist will help the gateway protect against malicious attacks from known illegal users.

[0065] The blacklist cache list is a dynamic blocking mechanism. When the crawler detection determines that the user or IP address has abnormal behavior patterns and abnormal access frequency, the system will add the user or IP address to the blacklist cache list. At the same time, the system will set a corresponding score value for each user in the blacklist cache list. If a user in the blacklist cache list is detected to log in to the gateway normally in the future, the system will reduce the score value corresponding to the user by one; if the user is detected to have malicious attacks on the gateway in the future, the system will increase the score value corresponding to the user by one. When the score value corresponding to the user is higher than the preset threshold, the system will add the user's identification information to the blacklist list. When the score value corresponding to the user is lower than the preset threshold, the system will delete the user's identification information from the blacklist cache list. Obviously, the dynamic change mechanism of the blacklist cache list can automatically respond to the malicious behavior of crawlers without manual intervention.

[0066] It should be noted that the creation process of the blacklist list and the blacklist cache list is completed by another server system, not by the gateway. The following describes three common crawler detection mechanisms to facilitate the understanding of the technical solutions provided by the present application by those skilled in the art. The three crawler detection mechanisms are: an anti-crawling mechanism based on real-time access, an anti-crawling mechanism based on traps, and an anti-crawling mechanism based on logs.

[0067] The anti-crawling mechanism based on real-time access is to add a layer of verification to the web site. Only after the request passes the verification can the backend logic be accessed. Since the mechanism needs to make an access judgment for each access request, in order not to have a great impact on the user's access, the judgment logic is relatively simple, so that when the web encounters high concurrent access, the user's access delay can also be within an acceptable range. This anti-crawling mechanism can directly obtain the data or perform simple calculations as the basis for judging the crawler when accessing the web. For example: the frequency of requests, the length of access time, etc. Common anti-crawling methods based on real-time access include: access rate detection method, User-Agent detection method, Referrer header detection method, client IP detection method, etc. Although the implementation principle of the anti-crawling mechanism based on real-time access is relatively simple, and it can also take quick response measures to the detected crawler behavior, the disadvantages of this mechanism are also obvious, that is, too much verification logic will increase the cost of the website and affect the user's access experience.

[0068] The anti-crawling mechanism based on trap technology is a way to actively protect the web from crawlers. This method can set links on the page that can be accessed by crawlers but not by users, and distinguish between crawler and user behavior by judging the way the link is triggered. The anti-crawling mechanism based on trap technology is mainly implemented through the following two methods:

[0069] First, a website with no actual value is specially built. This website is a big trap. Its main function is to be crawled and searched. Any behavior of visiting this website is suspicious. The crawler address library is established by collecting the access data of this website. However, this method has many disadvantages: for example, large resource consumption, poor real-time performance, etc.

[0070] Second, the web server provides a fake URL (Uniform Resource Locator) trap that is visible only to crawlers. Normal users will not enter the trap when browsing the web server, but crawlers will visit the trap page because they cannot distinguish between trap pages and normal pages. Therefore, users and crawlers can be distinguished by judging whether they visit the trap page. Under this setting, a more complex trap model can also be set up to construct a special infinite loop link page so that the pages that crawlers crawl after entering a trap are all traps, polluting the crawler's queue to be crawled and maliciously consuming crawler resources. For example, setting a picture with a size of zero on the page of the website cannot be observed by the naked eye, so people naturally cannot click on the picture, but the crawler can crawl the picture. This method can distinguish users from crawlers.

[0071] Although the anti-crawling mechanism based on trap technology can capture the crawlers that enter the trap in real time and quickly without relying on session information, the disadvantages of this mechanism are also obvious, that is, it lacks model support, is simple in layout, and is too rigid. Once it is discovered, targeted crawlers can be developed to avoid the trap. As time goes by, the traps set before are less and less effective, and more and more crawlers are able to bypass the mechanism.

[0072] In layman's terms, the log-based crawler detection mechanism simply processes the web access log and then statistically analyzes each access information in the access log. After a reasonable design, crawlers can be detected more accurately through comprehensive judgments in multiple dimensions without a large number of misjudgments. There is no impact on user access when judging crawlers.

[0073] After considering the three anti-crawling mechanisms, namely, the real-time access-based anti-crawling mechanism, the trap-based anti-crawling mechanism and the log-based anti-crawling mechanism, the final decision in this application is to use the log-based anti-crawling mechanism to create a blacklist list and a blacklist cache list.

[0074] When the gateway receives a target request, it first verifies the login environment of the target request to determine whether the login environment of the target request is safe and reliable. Specifically, the use of the reCAPTCHA human-machine verification service can effectively detect the user's login environment. If the score of the reCAPTCHA human-machine verification service is low, it means that the request is likely to be a crawler; if the score of the reCAPTCHA human-machine verification service is high, it means that the request is initiated by a real user and is a normal request. If the login environment of the target request fails the verification, it means that the login environment of the request is abnormal, and the gateway can refuse to respond to it.

[0075] When the login environment of the target request passes the verification, the identification information of the target request is obtained. Among them, the identification information of the target request includes the user name and IP address of the target request. Then, it is determined whether the identification information of the target request is in the blacklist. If the identification information of the target request is in the blacklist, it means that the target request is an illegal user. At this time, the gateway can refuse to respond to the target request.

[0076] If the identification information of the target request is not in the blacklist, the gateway will continue to determine whether the identification information of the target request is in the blacklist cache list. If the identification information of the target request is in the blacklist cache list, it means that the target request is an abnormal user. At this time, although the gateway will respond to the target request, it will prompt abnormal information.

[0077] Compared with the prior art, in this embodiment, the blacklist list and blacklist cache list created in advance can be used to constrain and limit the request to access the gateway, so the method can effectively limit the crawler behavior on the network and ensure the security of the network. In addition, the method is based on the asynchronous analysis of the historical log when analyzing the crawler behavior on the network, so that it will not affect the performance of normal users when accessing the network. At the same time, the historical log is deeply analyzed by the machine learning algorithm, so that the legitimacy of the historical request to access the gateway can be more accurately identified and analyzed, thereby ensuring the accuracy and reliability of the creation results of the blacklist list and the blacklist cache list, so that the accuracy of the identification of the crawler can be further improved. In addition, by designing the blacklist cache list, a certain cache mechanism is provided for possible misjudgment, which can not only reduce the impact of abnormal situations on the normal user access process, but also reduce the impact and intervention of human factors on the creation results of the blacklist cache list, so that the probability of misjudgment of network crawlers can be further reduced.

[0078] Based on the above embodiments, this embodiment further illustrates and optimizes the technical solution. As a preferred implementation, the process of creating a blacklist list and a blacklist cache list includes:

[0079] Perform crawler detection on historical logs to obtain target detection data;

[0080] Machine learning is performed on the target detection data to obtain target learning data, and a blacklist list and a blacklist cache list are created according to the target learning data.

[0081] See also Figure 2 , Figure 2The following is a diagram of creating a blacklist and a blacklist cache based on the historical logs of the gateway. When creating a blacklist and a blacklist cache, you need to obtain the logs corresponding to the client's request to access the gateway in advance and record these logs in the database DB. Then, perform crawler detection on the historical logs in the database DB to obtain target detection data. Finally, after performing machine learning on the target detection data and obtaining the target learning data, you can create a blacklist and a blacklist cache based on the target learning data. See Figure 3 , Figure 3 A schematic diagram of asynchronous storage of gateway historical logs.

[0082] After creating a blacklist list and a blacklist cache list based on the historical logs of the gateway, the blacklist list and the blacklist cache list can be used to effectively constrain and limit requests to access the gateway and ensure network security.

[0083] Obviously, through the technical solution provided in this embodiment, a blacklist list and a blacklist cache list can be created according to the historical logs of the gateway.

[0084] As a preferred implementation, the above step of performing crawler detection on historical logs to obtain target detection data includes:

[0085] The request frequency of the same user per unit time, the request time interval of the same user per unit time, the ratio of user agents being preset user agents, the ratio of requests carrying cookies, and the ratio of accessing static resources in historical logs are detected to obtain target detection data.

[0086] In layman's terms, the log-based crawler detection mechanism is to perform multi-dimensional analysis and processing on the historical logs of the gateway to determine whether a request is a crawler. Specifically, in this embodiment, whether a request is a crawler is mainly determined by the following five dimensions of indicators.

[0087] It is understandable that if a certain IP address sends a large number of requests within a certain period of time, and such requests are not allowed to be sent by normal users (for example: sending thousands of requests to the gateway within one minute), then it can be determined that the request is a crawler behavior.

[0088] Under normal circumstances, users' access to the gateway is not fixed, but often the crawler's access time interval is fixed or within a certain time range. For example, if the time interval between requests from the same IP address is within a fixed time interval, or within a time interval with a very small fluctuation range, then it can basically be determined that the request is a crawler behavior.

[0089] Generally speaking, each browser has a specific user agent (User-Agent). However, many crawlers are not designed with a user agent set. Therefore, if the user agent is not a common browser or a user agent with a programming language name, it can basically be determined that the request is a crawler behavior.

[0090] In addition, crawlers usually do not carry cookies when requesting pages, while normal users usually carry cookies from the previous page or the previous visit when accessing a page. Therefore, if a request does not carry the previous cookies, the request is most likely initiated by a crawler.

[0091] Crawlers also access static resources on the website, such as images, style sheets, etc. Normal users are less likely to access static resources. Therefore, if the frequency of access to static resources per unit time is high, it is very likely that the request is initiated by the crawler.

[0092] See also Figure 4 , Figure 4 This is a schematic diagram of content evaluation for crawler detection of gateway history logs. In practical applications, in order to crawl more effective information from the gateway history logs and better summarize the data detected by the crawler, you can also follow Figure 4 The content described is used to summarize and integrate the data detected from the historical logs of the gateway.

[0093] Specifically, the numbers 1, 2, 3, 4, and 5 can be used to represent the request frequency of the same user per unit time in the historical log, the request time interval of the same user per unit time, the ratio of the user agent being a preset user agent, the ratio of requests carrying cookies, and the ratio of accessing static resources. If the request frequency of the same user per unit time in the historical log is higher than the first preset threshold, it means that the probability of the user being a crawler is high; if the request frequency of the same user per unit time in the historical log is lower than the first preset threshold, it means that the probability of the user being a crawler is low. If the request time interval of the same user per unit time in the historical log is fixed, it means that the probability of the user being a crawler is high; if the request time interval of the same user per unit time in the historical log is not fixed, it means that the probability of the user being a crawler is low; if the ratio of the user agent being a preset user agent in the historical log is higher than the third preset threshold, it means that the probability of the user being a crawler is low; if the ratio of the user agent being a preset user agent in the historical log is lower than the third preset threshold, it means that the probability of the user being a crawler is high. If the ratio of requests carrying cookies in the history log is higher than the fourth preset threshold, it means that the probability of the user being a crawler is low; if the ratio of requests carrying cookies in the history log is lower than the fourth preset threshold, it means that the probability of the user being a crawler is high. If the ratio of requests to access static resources in the history log is high, it means that the probability of the user being a crawler is high; if the ratio of requests to access static resources in the history log is low, it means that the probability of the user being a crawler is low.

[0094] Obviously, through the technical solution provided in this embodiment, crawlers can be detected from the historical logs of the gateway.

[0095] As a preferred implementation, the above step of performing machine learning on the target detection data to obtain target learning data includes:

[0096] The decision tree algorithm is used to perform machine learning on the target detection data to obtain target learning data.

[0097] The decision tree algorithm is a basic classification and regression method. The decision tree model has a tree structure. In the classification problem, the tree structure represents the process of classifying instances based on features. The decision tree model can be considered as a collection of if-then rules, or as a conditional probability distribution defined in the feature space and class space. For example, the color, root and knocking sound can be considered as the three characteristics of a watermelon. Each decision is based on these three characteristics to divide a node into several new nodes.

[0098] See also Figure 5 , Figure 5This is a schematic diagram of the principle of the decision tree model. Since the structure of the decision tree algorithm is similar to a tree diagram, the decision process is clear and easy to understand and explain, which increases the convenience of machine learning for target detection data.

[0099] In this embodiment, when the analysis data of five dimensions, namely, the request frequency of the same user per unit time, the request time interval of the same user per unit time, the ratio of user agents being preset user agents, the ratio of requests carrying cookies, and the ratio of accessing static resources, are crawled out from the historical log of the gateway, and then the decision tree algorithm is used to perform machine learning on the extracted analysis data (target detection data), a blacklist list and a blacklist cache list can be created.

[0100] Obviously, through the technical solution provided in this embodiment, a blacklist list and a blacklist cache list can be created relatively quickly and conveniently.

[0101] Based on the above embodiments, this embodiment further illustrates and optimizes the technical solution. Figure 6 , Figure 6 A flowchart of another anti-crawler method based on energy management provided by an embodiment of the present invention. As a preferred implementation, the above step S11: when a target request is received, the login environment of the target request is verified, including:

[0102] Step S101: trigger the login page to perform human-machine verification on the login environment of the target request, and determine whether the login environment of the target request passes the verification; if so, execute step S102; if not, execute step S104;

[0103] Step S102: Feedback the environment verification code to the login page, so that the target request carries the environment verification code to log in to the gateway;

[0104] Step S103: When a target request carrying an environment verification code is received, the environment verification code is verified and it is determined whether the environment verification code passes the verification; if so, step S12 is continued.

[0105] In this embodiment, when the gateway verifies the login environment of the target request, it first triggers the login page to perform a human-machine verification on the login environment of the target request. If the login environment of the target request passes the human-machine verification of the login page, the gateway feeds back the environment verification code to the login page, so that the target request can log in to the gateway with the environment verification code. When the gateway receives a target request carrying an environment verification code, it needs to verify the environment verification code. If the environment verification code passes the verification, step S12 is continued.

[0106] It should be noted that in this application, the login page is at the front end and the gateway is at the back end. The gateway is equivalent to the back end of the login page.

[0107] As a preferred implementation, the above step 101: triggering the login page to perform human-machine verification on the login environment of the target request, and determining whether the login environment of the target request passes the verification, further includes:

[0108] Step S104: Feedback the behavior verification code to the login page so that the login page can use the behavior verification code to verify the login environment of the target request, and determine whether the login environment of the target request passes the verification of the behavior verification code; if so, continue to step S102; if not, continue to step S104.

[0109] If the login environment of the target request does not pass the verification of the login page, the gateway will feedback the behavior verification code to the login page, so that the login page can use the behavior verification code to verify the login environment of the target request. If the login environment of the target request passes the verification of the behavior verification code, then step S102 will be continued; if the login environment of the target request does not pass the verification of the behavior verification code, then step S104 will be continued.

[0110] See also Figure 7 , Figure 7 Schematic diagram of three types of behavior verification codes provided by embodiments of the present invention. Behavior verification codes are a popular type of verification code. Behavior verification codes require users to perform some additional operations to prove that the request is initiated by a real user, not a robot. Behavior verification codes complete verification through user operation behaviors without having to read distorted pictures and text.

[0111] Obviously, through the technical solution provided in this embodiment, the accuracy and reliability of the gateway in verifying the login environment of the target request can be further improved.

[0112] As a preferred implementation, the above step of: after determining whether the identification information of the target request is in the blacklist cache list, further includes:

[0113] The blacklist cache list is updated according to the identification information of the target request.

[0114] In this embodiment, the blacklist cache list can also be updated according to the identification information of the target request. Specifically, a corresponding score value can be set in advance for the identification information of each user stored in the blacklist cache list. If the identification information of the target request is in the blacklist cache list, the score value corresponding to the identification information of the target request is increased by one; if the identification information of the target request is not in the blacklist cache list, the score value corresponding to the identification information of the target request is decreased by one.

[0115] In the process of the gateway receiving requests in real time, the score value of the identification information corresponding to each user stored in the blacklist cache list can be cumulatively added and / or cumulatively subtracted according to the above method. If the score value of the identification information corresponding to a certain user in the blacklist cache list is greater than the preset threshold, the identification information of the user can be added to the blacklist list. If the score value of the identification information corresponding to a certain user in the blacklist cache list is less than the preset threshold, the identification information of the user can be removed from the blacklist cache list.

[0116] Obviously, through the technical solution provided by this embodiment, it can be ensured that the information stored in the blacklist cache list is compatible with the real-time status of the request received by the gateway.

[0117] See also Figure 8 , Figure 8 A structural diagram of an anti-crawler device based on energy management provided by an embodiment of the present invention, the device includes:

[0118] The environment verification module 21 is used to verify the login environment of the target request when receiving the target request;

[0119] The identification acquisition module 22 is used to obtain the identification information of the target request if the login environment of the target request passes the verification, and determine whether the identification information of the target request is in the blacklist;

[0120] A request response module 23, configured to refuse to respond to the target request when the determination result of the identification acquisition module is yes;

[0121] A list judgment module 24 is used to judge whether the identification information of the target request is in a blacklist cache list when the judgment result of the identification acquisition module is no; wherein the blacklist list and the blacklist cache list are lists created based on a machine learning algorithm and according to historical logs; the blacklist list stores the identification information of illegal users, and the blacklist cache list stores the identification information of abnormal users; the historical log is a log corresponding to historical requests for accessing the gateway;

[0122] The information prompt module 25 is used to respond to the target request and prompt abnormal information when the determination result of the list determination module is yes.

[0123] Preferably, the list determination module 24 includes:

[0124] A data detection unit, used to perform crawler detection on the historical log to obtain target detection data;

[0125] A list creation unit is used to perform machine learning on the target detection data to obtain target learning data, and to create the blacklist list and the blacklist cache list according to the target learning data.

[0126] Preferably, the data detection unit includes:

[0127] The data detection subunit is used to detect the request frequency of the same user per unit time, the request time interval of the same user per unit time, the ratio of user agents being preset user agents, the ratio of requests carrying cookies, and the ratio of accessing static resources in the historical log to obtain the target detection data.

[0128] Preferably, the list creation unit includes:

[0129] The machine learning subunit is used to perform machine learning on the target detection data using a decision tree algorithm to obtain the target learning data.

[0130] Preferably, the environment verification module 21 includes:

[0131] An environment verification unit, used to trigger the login page to perform a human-machine verification on the login environment of the target request, and determine whether the login environment of the target request passes the verification;

[0132] A verification code feedback unit, configured to feedback an environment verification code to the login page when the determination result of the environment verification unit is yes, so that the target request carries the environment verification code to log in to the gateway;

[0133] A verification code verification unit, configured to verify the environment verification code upon receiving the target request carrying the environment verification code, and determine whether the environment verification code passes the verification;

[0134] The step jump unit is used to continue to execute the step of obtaining the identification information of the target request and judging whether the identification information of the target request is in the blacklist when the judgment result of the verification code verification unit is yes.

[0135] Preferably, it also includes:

[0136] a behavior code judgment unit, configured to feed back a behavior verification code to the login page when the judgment result of the environment verification unit is negative, so that the login page verifies the login environment of the target request by using the behavior verification code, and determines whether the login environment of the target request passes the verification of the behavior verification code;

[0137] A first jump unit, configured to, when the determination result of the behavior code determination unit is yes, continue to execute the step of feeding back the environment verification code to the login page so that the target request carries the environment verification code to log in to the gateway;

[0138] The second jump unit is used to continue to execute the step of feeding back the behavior verification code to the login page when the judgment result of the behavior code judgment unit is no, so that the login page uses the behavior verification code to verify the login environment of the target request, and judge whether the login environment of the target request passes the verification of the behavior verification code.

[0139] Preferably, it also includes:

[0140] The list updating module is used to update the blacklist cache list according to the identification information of the target request after determining whether the identification information of the target request is in the blacklist cache list.

[0141] An anti-crawler device based on energy management provided by an embodiment of the present invention has the beneficial effects of the anti-crawler method based on energy management disclosed above.

[0142] See also Fig. 9 , Fig. 9 A structural diagram of an electronic device provided by an embodiment of the present invention, the electronic device comprising:

[0143] A memory 31, used for storing computer programs;

[0144] The processor 32 is used to implement the steps of an anti-crawler method based on energy management as disclosed above when executing the computer program.

[0145] An electronic device provided by an embodiment of the present invention has the beneficial effects of the anti-crawler method based on energy management disclosed above.

[0146] Correspondingly, an embodiment of the present invention further provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of an anti-crawler method based on energy management as disclosed above.

[0147] A computer program product provided by an embodiment of the present invention has the beneficial effects of the aforementioned anti-crawler method based on energy management.

[0148] Correspondingly, an embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of an anti-crawler method based on energy management as disclosed above are implemented.

[0149] A computer-readable storage medium provided by an embodiment of the present invention has the beneficial effects of the aforementioned anti-crawler method based on energy management.

[0150] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0151] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0152] The above is a detailed introduction to the anti-crawler method, device, equipment, program product and medium based on energy management provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. An anti-crawler method based on energy management, characterized in that: Applied to gateways, including: When receiving a target request, determining whether the login environment of the target request passes the verification; If yes, obtaining the identification information of the target request, and determining whether the identification information of the target request is in the blacklist; If so, refusing to respond to the target request; If not, determine whether the identification information of the target request is in the blacklist cache list; wherein the blacklist list and the blacklist cache list are lists created based on a machine learning algorithm and according to historical logs; the blacklist list stores the identification information of illegal users, and the blacklist cache list stores the identification information of abnormal users; the historical log is a log corresponding to historical requests for accessing the gateway; If yes, respond to the target request and prompt abnormal information.

2. The anti-crawler method based on energy management according to claim 1, characterized in that: The process of creating the blacklist list and the blacklist cache list includes: Perform crawler detection on the historical log to obtain target detection data; Machine learning is performed on the target detection data to obtain target learning data, and the blacklist list and the blacklist cache list are created according to the target learning data.

3. The anti-crawler method based on energy management according to claim 2, characterized in that: The crawler detection of the historical log to obtain target detection data includes: The target detection data is obtained by detecting the request frequency of the same user per unit time, the request time interval of the same user per unit time, the ratio of user agents being preset user agents, the ratio of requests carrying cookies, and the ratio of accessing static resources in the historical log.

4. The anti-crawler method based on energy management according to claim 2, characterized in that: The performing machine learning on the target detection data to obtain target learning data includes: The target detection data is subjected to machine learning using a decision tree algorithm to obtain the target learning data.

5. The anti-crawler method based on energy management according to claim 1, characterized in that: When receiving the target request, the login environment of the target request is verified, including: Triggering the login page to perform a human-machine verification on the login environment of the target request, and determining whether the login environment of the target request passes the verification; If yes, feeding back the environment verification code to the login page, so that the target request carries the environment verification code to log in to the gateway; When receiving the target request carrying the environment verification code, verifying the environment verification code and determining whether the environment verification code passes the verification; If so, continue to execute the step of obtaining the identification information of the target request and determining whether the identification information of the target request is in the blacklist.

6. The anti-crawler method based on energy management according to claim 5, characterized in that: After the trigger login page performs a human-machine verification on the login environment of the target request and determines whether the login environment of the target request passes the verification, the method further includes: If not, feeding back a behavior verification code to the login page, so that the login page uses the behavior verification code to verify the login environment of the target request, and determines whether the login environment of the target request passes the verification of the behavior verification code; If yes, continue to execute the step of feeding back the environment verification code to the login page so that the target request carries the environment verification code to log in to the gateway; If not, continue to execute the step of feeding back the behavior verification code to the login page, so that the login page uses the behavior verification code to verify the login environment of the target request, and determine whether the login environment of the target request passes the verification of the behavior verification code.

7. An anti-crawler method based on energy management according to any one of claims 1 to 6, characterized in that: After determining whether the identification information of the target request is in the blacklist cache list, the method further includes: The blacklist cache list is updated according to the identification information of the target request.

8. An anti-crawler device based on energy management, characterized in that: Applied to gateways, including: An environment verification module, used for verifying the login environment of the target request when receiving the target request; An identification acquisition module, used for acquiring identification information of the target request if the login environment of the target request passes the verification, and determining whether the identification information of the target request is in the blacklist; A request response module, configured to refuse to respond to the target request when the determination result of the identification acquisition module is yes; A list judgment module, used for judging whether the identification information of the target request is in a blacklist cache list when the judgment result of the identification acquisition module is no; wherein the blacklist list and the blacklist cache list are lists created based on a machine learning algorithm and according to historical logs; the blacklist list stores the identification information of illegal users, and the blacklist cache list stores the identification information of abnormal users; the historical log is a log corresponding to historical requests for accessing the gateway; The information prompt module is used to respond to the target request and prompt abnormal information when the determination result of the list determination module is yes.

9. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, used to implement the steps of an anti-crawler method based on energy management as described in any one of claims 1 to 7 when executing the computer program.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of an anti-crawler method based on energy management as described in any one of claims 1 to 7 are implemented.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of an anti-crawler method based on energy management as described in any one of claims 1 to 7 are implemented.

Images