Anti-crawler method and device, electronic equipment, storage medium and computer program product

Abstract

The invention provides an anti-crawler method and device, electronic equipment, a storage medium and a computer program product, and the method comprises the steps: generating one or more website files corresponding to a first website based on a source code of the first website according to a set generation rule, and generating one or more website files corresponding to the first website for each website file in the one or more website files; starting a corresponding network service; wherein the set generation rule is used for modifying a positioning path of an element in the source code; the network service is used for calling a corresponding website file to respond to a user request.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to an anti-crawler method, apparatus, electronic device, storage medium, and computer program product. Background Technology

[0002] Currently, in anti-crawler techniques that use Cascading Style Sheets (CSS) offset technology to adjust the position of webpage content, the CSS offset rules are fixed settings, which are easily cracked by crawler programs, threatening the security of webpage content. If the CSS offset rules are designed to be too complex and difficult to crack, it will increase the difficulty of page rendering and affect page rendering performance. Summary of the Invention

[0003] To address the related technical issues, embodiments of this application provide an anti-crawler method, apparatus, electronic device, storage medium, and computer program product.

[0004] The technical solution of this application embodiment is implemented as follows: This application provides an anti-crawler method, the method comprising: According to the set generation rules, based on the source code of the first website, one or more website files corresponding to the first website are generated, and for each of the one or more website files, a corresponding network service is started; wherein, the set generation rules are used to modify the location path of elements in the source code; the network service is used to call the corresponding website file to respond to user requests.

[0005] In the above scheme, the setting generation rules include one or more of the following: Replace the names of the element attributes in the source code with random gibberish; Invalid element tags are randomly inserted into the source code.

[0006] The method in the above scheme further includes: Receive user requests sent by the client; The user request is forwarded to a network service in the network service list that meets the first set condition, and the corresponding website file is obtained; The corresponding website file is returned to the client.

[0007] In the above scheme, the first setting condition includes one or more of the following: The total frequency of received user requests is less than or equal to a first set threshold. The startup time is less than or equal to the second set threshold; The number of user requests currently received is less than or equal to the third set threshold; The corresponding weight is the highest, and the weight of network services is determined based on the total frequency of user requests received.

[0008] The method in the above scheme further includes: If a network service that meets the second set condition exists in the network service list, a first operation is performed; the first operation is used to update the network service list.

[0009] In the above scheme, the second setting condition includes one or more of the following: The startup duration is greater than or equal to the fourth set threshold; The total frequency of received user requests is greater than or equal to the fifth set threshold; The frequency of receiving user requests from the same client is greater than or equal to the sixth set threshold.

[0010] In the above scheme, the first operation includes one or more of the following: Stop network services that meet the second set condition; Delete the website files corresponding to the network services that meet the second set conditions; Stop forwarding user requests to network services that meet the second set condition; Determine whether the waiting time of the network service that meets the second set condition is greater than or equal to the seventh set threshold, wherein the waiting time represents the time interval between the last time the network service received the user request and the current time. According to the established generation rules, based on the source code of the first website, a set number of website files corresponding to the first website are regenerated. For each website file in the set number of website files, start the corresponding network service.

[0011] This application also provides an anti-crawler device, including: The generation unit is used to generate one or more website files corresponding to the first website based on the source code of the first website, according to the set generation rules. The startup unit is used to start a corresponding network service for each of the one or more website files; wherein the set generation rule is used to modify the location path of the element in the source code; the network service is used to call the corresponding website file to respond to user requests.

[0012] This application also provides an electronic device, including a processor and a memory for storing a computer program that can run on the processor, wherein the processor, when running the computer program, performs the steps of any of the methods described above.

[0013] This application also provides a storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of any of the above methods.

[0014] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the above methods.

[0015] The anti-crawler methods, devices, electronic devices, storage media, and computer program products provided in this application embodiment generate one or more website files corresponding to the first website based on the source code of the first website according to a set generation rule, and start a corresponding network service for each of the one or more website files; wherein, the set generation rule is used to modify the positioning path of elements in the source code; the network service is used to call the corresponding website file to respond to user requests. The above scheme, by setting generation rules, allows the server to modify the positioning path of web page elements in the source code, making it difficult for crawler programs to locate web page elements, preventing the target content from being obtained by the crawler program, thus limiting the crawler program's ability to locate the target content, making it difficult for the crawler program to perform automated and batch crawling, and improving the security of web page data. Attached Figure Description

[0016] Figure 1 A flowchart illustrating an anti-crawler method provided in an embodiment of this application; Figure 2 An example diagram of an anti-crawler system architecture is provided for application embodiments of this application; Figure 3 This is a schematic diagram of the structure of an anti-crawler device provided in an embodiment of this application; Figure 4 This is a schematic diagram of the electronic device structure provided in the embodiments of this application. Detailed Implementation

[0017] Web crawlers are a common type of web program used to automatically acquire information and data on the internet. Malicious web crawlers can put enormous pressure on target website servers by scraping information and data in batches, at high frequency, and automatically. They can also cause security problems such as data abuse and privacy breaches. Therefore, anti-crawler measures are an important research area in the security field.

[0018] The common process of web crawling tools includes: analyzing the page to find the desired data content and the page's Uniform Resource Locator (URL); locating the HTML element tag containing the data content using Cascading Style Sheets (CSS) selector paths or XML Path Language (XPath) paths; copying and storing the data, and then automatically redirecting to the next target page; repeating the above steps.

[0019] Current anti-crawler technologies generally include the following: 1) Calculating user access frequency based on the user's Internet Protocol (IP) to determine whether it is crawler traffic. If it is crawler traffic, rate limiting or access restriction is implemented; 2) Determining whether it is crawler traffic based on information such as request header fields and small text file fields (such as cookies). If it is crawler traffic, rate limiting or access restriction is implemented; 3) Performing human-machine identification verification through CAPTCHAs; 4) Using CSS offset technology to adjust the content position to achieve anti-crawler measures.

[0020] Crawlers are designed to automatically obtain useful information on web pages. Web page content consists of HTML element tags and CSS styles. HTML elements contain the data part of the web page, and CSS styles are used to set the styles of web page HTML elements, such as position, color, size, etc. Based on this, the anti-crawling technology that adjusts the content position through CSS offset technology means that the website makes the HTML elements store disordered data content through certain rules, and then uses CSS styles to rearrange the disordered text into the normal reading order of humans. As a result, the web page content automatically obtained by the crawler is disordered and unusable information, while the content seen by users on the web page is the data content rearranged by CSS styles, which is ordered and useful information. In practical applications, the rules of CSS offset may vary, but in order to allow users to browse the web page normally, the rules of CSS offset for a unified website are usually fixed and difficult to modify. If the other party calculates the CSS offset amount through analysis and adds the anti-offset operation to the crawler program according to the CSS offset amount to reverse process the crawled data, it is possible to bypass the CSS offset anti-crawling restriction and obtain the correct information content. For example, in the HTML element, there are two text contents, "yun dan feng feng bu" and "sha neng li zhong xin", which are disordered texts. The crawler program cannot obtain useful information. The CSS offset rule is to use several characters at the end of the second line of text to cover the corresponding characters in the first line of text. That is, use the last 4 characters "neng li zhong xin" in the second line of text "sha neng li zhong xin" to cover the last four characters "dan feng feng bu" in the first line of text "yun dan feng feng bu". When the web page is displayed after CSS offset settings, the ordered text "yun neng li zhong xin" can be displayed on the web page.

[0021] Therefore, although using CSS offset can obfuscate web page content, restrict crawlers from obtaining valid content on the page, that is, target information, and increase the difficulty of crawlers, the crawler program can still obtain disordered data containing target information and is easy to analyze the CSS offset rule, that is, it is easy to perform batch anti-offset processing on the crawled data to obtain the target information. The CSS offset method has a weak restriction on crawlers. If the complexity of the CSS offset rule is increased to improve the difficulty of the crawler program in analyzing the CSS offset rule, it will increase the complexity of web page data, affecting page rendering performance and user experience.

[0022] Based on this, in various embodiments of this application, according to a set generation rule, one or more website files corresponding to the first website are generated based on the source code of the first website, and a corresponding network service is started for each of the one or more website files; wherein, the set generation rule is used to modify the positioning path of elements in the source code; the network service is used to call the corresponding website file to respond to user requests. The above scheme, by setting generation rules, allows the server to modify the positioning path of web page elements in the source code, making it difficult for the crawler to locate the web page elements, preventing the target content from being obtained by the crawler, thus limiting the crawler's ability to locate the target content, making it difficult for the crawler to perform automated and batch crawling, and improving the security of web page data.

[0023] The present application will now be described in further detail with reference to the accompanying drawings and embodiments.

[0024] This application provides an anti-crawler method applied to a server-side electronic device, such as a server, which provides World Wide Web services. Figure 1 As shown, the method includes: Step 101: According to the set generation rules, based on the source code of the first website, generate one or more website files corresponding to the first website, and start the corresponding network service for each of the one or more website files.

[0025] The set generation rules are used to modify the location path of elements in the source code; the network service is used to call the corresponding website file to respond to user requests.

[0026] Here, it's possible to pre-generate and store one or more website files corresponding to the first website according to pre-defined generation rules, and then randomly return one website file from the one or more website files corresponding to the user's request upon receiving a user request; alternatively, upon receiving a user request, one or more website files corresponding to that user request are generated according to pre-defined generation rules, and one website file is randomly returned from the one or more website files corresponding to that user request. When generating one or more website files, a corresponding web service is started for each website file. The web service corresponding to each website file in the one or more website files can be placed in a web service list for unified management. The web service list is used to manage currently available web services. Setting generation rules can also be understood as anti-crawler operations.

[0027] It should be noted that the generation rules can adhere to one or more of the following principles: 1. The display effect and user behavior logic of each website file in the browser are completely consistent with the display effect and user behavior logic of the source code in the browser, that is, setting the generation rules will not affect the normal use of the website; 2. In different website files generated according to the set generation rules, the CSS selector path of the same web page element is different, and the XPath path of the same web page element is also different.

[0028] It's important to note that the first website can be any website within the server. Each of the one or more website files contains all the information needed to respond to any user request, and each website file can independently handle user requests. For example, if the source code is written based on a single-page framework, the server can return all files or source code of the first website and, according to the set generation rules, generate one or more website files based on those files or source code. As another example, if the source code is written based on a multi-page framework, the server only needs to return the file or source code of the page requested by the user and, according to the set generation rules, generate one or more website files based on that page's file or source code. The location paths of web page elements within different website files corresponding to the first website are different. Website files can also be understood as web page resources, including HTML files, CSS files, JS (JavaScript) files, and other static resource files. Web services, also known as network services, are used to provide external interfaces for the corresponding website files, that is, to provide web services related to the corresponding website files. A website can also be understood as a web application.

[0029] Source code can represent the initial native code written by website developers, or it can represent the code after the native code has been bundled and compressed using a bundling tool (such as Webpack). It can include HTML files that constitute the main content of the website, CSS files that style the element structure within the HTML files, JS files that implement user interaction between the webpage and the user, and other static resource files; other static resource files include media files, such as images. Generally, the source code of the same version of a website will not change, meaning the HTML element structure and attributes, and the CSS files will not change. Therefore, the syntax of the CSS selector paths or XPath paths of the webpage elements within the website usually remains the same. Thus, web crawlers can easily locate the corresponding webpage elements based on fixed and standardized CSS selector paths or XPath paths. In this embodiment, by setting generation rules to modify the CSS selector paths or XPath paths of the webpage elements in the source code, the CSS selector paths or XPath paths of the webpage elements can be dynamically changed. This prevents web crawlers from locating the corresponding webpage elements using fixed and standardized CSS selector paths or XPath paths, achieving the purpose of anti-crawling.

[0030] To increase the difficulty of cracking the location path of web page elements, in one embodiment, the set generation rule includes one or more of the following: Replace the names of the element attributes in the source code with random gibberish; Invalid element tags are randomly inserted into the source code.

[0031] Here, the names of element attributes in the source code are replaced with random gibberish. Specifically, by traversing the source code, when element attributes are identified, such as the id and class attributes of HTML elements, the attribute names are replaced with random gibberish to change the CSS selector path of the HTML elements. This prevents the crawler from locating elements through CSS selectors, including class selectors and id selectors. The random gibberish can be a random string, and the number of characters in the random string can also be randomly determined. The length of the random string can be limited by setting a character limit. For example, if two class attributes, named "data" and "data2", are identified in the source code and used to locate target element 1 and target element 2 respectively, "data" is replaced with random gibberish "fjhfeeb364", and "data2" is replaced with random gibberish "mnr347sgwh". This prevents the crawler from directly locating elements through the class selectors of the originally named "date" and "data2". The random garbled characters for the same element attribute name are different in different website files, meaning that the randomly generated garbled characters are different each time. This increases the randomness of the element attribute name and the randomness of the website files returned for user requests, making it more difficult for web crawlers to crack and further improving the security of page information or web page data.

[0032] Randomly insert invalid element tags into the source code; invalid element tags can be understood as element tags that are not displayed or hidden on the page, or meaningless tags; element tags that are not displayed or hidden on the page can be set by tag styles, for example, setting the embedded style of the element tag to style="display:none"; by randomly inserting invalid element tags; by inserting invalid element tags, the DOM tree hierarchy can be increased or the same layer tag sequence can be changed, thereby changing the xpath path of the HTML element, so as to prevent the crawler program from locating the element through a fixed xpath path. For example, in the source code, a web page element can be located by the xpath path " / html / body / div[1]", and by randomly inserting invalid element tags into the source code, it can be added before the web page element. <customer-name> Added later< / customer-name> This can be understood as adding a parent node, customer-name, to the webpage element in the DOM tree, which increases the level of the DOM tree. After insertion, the xpath path of the webpage element becomes " / html / body / customer-name / div[1]"; it can also be inserted before the webpage element: Randomly obfuscated content, page not displayed.The text content is not displayed on the page and is hidden, which means that the same layer tag sequence is changed. After insertion, the XPath path of the web page element becomes " / html / body / customer-name / div[2]". The XPath path of the same element in different website files is different, that is, the content and / or position of the invalid element tag randomly inserted each time is different, which increases the randomness of the XPath path and the randomness of the website file returned for the user's request, increases the difficulty of the crawler program to crack, and further improves the security of the page information.

[0033] In order to prevent web scraping without affecting the normal use of the website, in one embodiment, the method further includes: Receive user requests sent by the client; The user request is forwarded to a network service in the network service list that meets the first set condition, and the corresponding website file is obtained; The corresponding website file is returned to the client.

[0034] Here, upon receiving a user request from a client, the system can either forward the request to a pre-generated network service in the network service list that meets a first set condition, to obtain the corresponding website file; or trigger the generation of a website file, i.e., generate one or more website files corresponding to the user request based on the source code of the response file corresponding to the user request according to set generation rules, or generate one or more website files corresponding to a first website based on the source code of a first website. For each website file, a corresponding network service is started, and the network service corresponding to each website file can be added to the network service list. Then, the user request is sent to the network service in the network service list that meets the first set condition to obtain the corresponding website file. The obtained corresponding website file is then returned to the client to respond to the user request sent by the client, allowing the client to parse the website file and render the corresponding page on the display for the user to view. A network service meeting the first set condition indicates that the network service currently has high security and can continue to provide web services and process user requests.

[0035] To further prevent web crawlers from maliciously scraping web page information and improve the security of web page information, in one embodiment, the first setting condition includes one or more of the following: The total frequency of received user requests is less than or equal to a first set threshold. The startup time is less than or equal to the second set threshold; The number of user requests currently received is less than or equal to the third set threshold; The corresponding weight is the highest, and the weight of network services is determined based on the total frequency of user requests received.

[0036] Here, for each network service in the network service list, we can determine whether the network service meets the first set condition by analyzing its startup time and the number of user requests received. The total frequency of received user requests can be the total number of user requests received within a certain period, which can be a set statistical time unit, such as 1 hour. The startup time of the network service can be understood as the service duration. The number of currently received user requests can be used to assess the load of the network service. The fewer the number of currently received user requests, the fewer user requests the network service needs to handle, and the lower the load of the network service. The more the number of currently received user requests, the more user requests the network service needs to handle, and the higher the load of the network service. The weight of the network service can be determined based on the total frequency of received user requests. For example, the reciprocal of the total frequency of received user requests can be used as the weight of the network service. That is, the higher the total frequency of received user requests, the lower the weight of the network service, and the lower the total frequency of received user requests, the higher the weight of the network service.

[0037] The first set condition may include one or more of the following: the total frequency of received user requests is less than or equal to a first set threshold; the startup time is less than or equal to a second set threshold; the number of currently received user requests is less than or equal to a third set threshold; and the corresponding highest weight. For example, the first set condition may include the total frequency of received user requests being less than or equal to the first set threshold and the startup time being less than or equal to the second set threshold. A network service that meets the first set condition is a network service whose total frequency of received user requests is less than or equal to the first set threshold and whose startup time is less than or equal to the second set threshold.

[0038] A high frequency of receiving user requests and / or a long startup time will increase the likelihood that the anti-crawler processing in the network service can be cracked. On the other hand, a high load on the network service will affect the efficiency of user request processing. In this embodiment, user requests are forwarded to network services that meet one or more of the first set conditions. This ensures that the network service that processes user requests receives user requests at a low frequency and / or has a short startup time, thereby reducing the likelihood that the anti-crawler processing in the network service can be cracked, improving the security of network information, and ensuring the efficiency of user request processing.

[0039] To enhance the dynamic changes and flexibility of network services, thereby strengthening anti-crawler effects and improving the security of web page information, in one embodiment, the method further includes: If a network service that meets the second set condition exists in the network service list, a first operation is performed; the first operation is used to update the network service list.

[0040] Here, a network service meeting the second set condition indicates a high probability that the anti-crawling operations within the website files corresponding to that network service can be cracked. If a network service in the network service list meets the second set condition, a first operation is performed on that service, such as deleting or resetting the network service in the list to reduce the risk of the anti-crawling operations within the website files corresponding to that service being cracked.

[0041] In order to remove high-risk online services and improve the security of online services in the online service list, that is, to improve the security of web page information, in one embodiment, the second setting condition includes one or more of the following: The startup duration is greater than or equal to the fourth set threshold; The total frequency of received user requests is greater than or equal to the fifth set threshold; The frequency of receiving user requests from the same client is greater than or equal to the sixth set threshold.

[0042] Here, if the startup time of a network service is greater than or equal to the fourth preset threshold, and / or the total frequency of user requests received by the network service is greater than or equal to the fifth preset threshold, it indicates that the anti-crawler operations within the website files corresponding to the network service are at risk of being cracked; if the frequency of user requests received by the network service from the same client is greater than or equal to the sixth preset threshold, that is, if the same client requests the website files corresponding to the network service multiple times, the client is very likely to be a crawler program, and after making multiple requests to the website files corresponding to the network service, the client is very likely to analyze the anti-crawler operations within the website files obtained multiple times, thereby being able to bypass the anti-crawler operations and obtain the correct web page information; therefore, network services that meet the second preset condition need to be dynamically updated.

[0043] To achieve dynamic updates to the network service list and improve the security of the network services in the list, i.e., to enhance the security of webpage information, in one embodiment, the first operation includes one or more of the following: Stop network services that meet the second set condition; Delete the website files corresponding to the network services that meet the second set conditions; Stop forwarding user requests to network services that meet the second set condition; Determine whether the waiting time of the network service that meets the second set condition is greater than or equal to the seventh set threshold, wherein the waiting time represents the time interval between the last time the network service received the user request and the current time. According to the established generation rules, based on the source code of the first website, a set number of website files corresponding to the first website are regenerated. For each website file in the set number of website files, start the corresponding network service.

[0044] Here, for network services that meet the second set condition, i.e., network services whose anti-crawler operations within the corresponding website files are at risk of being cracked, the website files corresponding to the network service can be deleted in a timely manner, and / or the network service can be stopped. Alternatively, user requests can be stopped from being forwarded to network services that meet the second set condition before deleting the network files corresponding to the network service and / or stopping the network service. In the case of stopping the forwarding of user requests to network services that meet the second set condition, it can also be determined whether the waiting time of the network service that meets the second set condition is greater than or equal to the seventh set threshold. If the waiting time of the network service that meets the second set condition is greater than or equal to the seventh set threshold, then the network files corresponding to the network service can be deleted and / or the network service can be stopped. By setting a waiting time as a buffer instead of immediately deleting the network files corresponding to the network service and / or stopping the network service, it can be ensured that the currently received user requests can be processed, avoiding affecting the normal user request response process and impacting the user experience.

[0045] Even after stopping network services that meet the second set conditions and deleting the corresponding website files, the system can still regenerate a set number of website files corresponding to the first website based on the source code of the first website, according to the set generation rules. For each website file in the set number of website files, the corresponding network service will be started, thereby enabling dynamic updates of network services and website files. This achieves dynamic updates of anti-crawler operations, preventing fixed anti-crawler operations from being easily cracked by crawler programs. In other words, this anti-crawler method has high availability. Dynamic updates also prevent attacks and tampering of a particular network service from affecting user request responses, providing anti-tampering capabilities. One or more network services can meet the second set conditions; the set number can be the same as the number of network services meeting the second set conditions, or it can be determined by setting a network service upper limit.

[0046] The following is based on Figure 2Taking the anti-crawler system architecture shown as an example, this application will be further described in detail with application examples. Arrows represent the direction of information transmission; the generator is used to generate website files and start network services. Specifically, it can be used to perform obfuscation operations on the source code of the first website, such as renaming attribute names and randomly inserting invalid HTML elements. These obfuscation operations can also be understood as anti-crawler operations, generating website files so that web page elements cannot be located by fixed CSS selector paths or XPath paths, preventing crawlers from locating and crawling web page content; the generation controller is used to manage or update the network service list and website files. Specifically, it can be used to control the generation and deletion of website files, as well as the start and stop of network services, maintaining continuous updates to website files and corresponding network services, preventing crawlers from bypassing anti-crawler operations and automating and batch acquiring web page content, thus increasing server load; the forwarding controller is used to forward user requests to network services and record the forwarding status of user requests, providing data for updating the network service list and ensuring the stability and reliability of normal user access. The generator, generation controller, and forwarding controller can all be independent devices or deployed on the same device. The forwarding controller and generation controller can be merged. The specific architecture can be determined according to the actual application and is not specifically limited here.

[0047] Anti-scraping methods may include the following steps: Step 1: According to the set generation rules, based on the source code of the first website, generate one or more website files corresponding to the first website, and start the corresponding network service for each website file in the one or more website files.

[0048] Here, setting generation rules can include replacing element attribute names in the source code with random gibberish, and / or randomly inserting invalid element tags into the source code. Specific implementation methods can be found in the relevant descriptions above, and will not be elaborated upon here. Figure 2 In the architecture, step 3 is executed by the generator.

[0049] It should be noted that the initial number of website files can be determined based on the set initial quantity. Figure 2 In the architecture, the generation controller can determine the initial quantity and send the initial quantity to the generator.

[0050] Step 2: Receive user requests sent by the client.

[0051] Here, the client can send user requests to the server through a browser. Figure 2 In this architecture, the forwarding controller can receive user requests sent by the client.

[0052] Step 3: Forward the user request to the network service in the network service list that meets the first set condition, and obtain the corresponding website file.

[0053] Here, the first set of conditions includes one or more of the following: the total frequency of received user requests is less than or equal to a first set threshold; the startup time is less than or equal to a second set threshold; the number of currently received user requests is less than or equal to a third set threshold; and the service with the highest corresponding weight. When multiple network services meet the first set of conditions, user requests can be randomly forwarded to any network service that meets the first set of conditions. Figure 2 In this architecture, the forwarding controller can forward user requests to network services that meet the first set conditions.

[0054] Step 4: Return the corresponding website files to the client.

[0055] Here, when a user request is forwarded to a network service in the network service list that meets the first set condition, the network service, based on the received user request, calls the corresponding website file to respond to the user request, and then returns the corresponding website file to the client, so that the client can parse the website file and render the corresponding page on the display for the user to view.

[0056] Step 5: If a network service that meets the second set condition exists in the network service list, perform the first operation.

[0057] Here, the second set of conditions includes one or more of the following: startup duration greater than or equal to the fourth set threshold; total frequency of received user requests greater than or equal to the fifth set threshold; and frequency of received user requests from the same client greater than or equal to the sixth set threshold. The first operation includes one or more of the following: stopping network services that meet the second set of conditions; deleting website files corresponding to network services that meet the second set of conditions; stopping forwarding user requests to network services that meet the second set of conditions; determining whether the waiting time of network services that meet the second set of conditions is greater than or equal to the seventh set threshold; regenerating a set number of website files corresponding to the first website based on the source code of the first website according to the set generation rules; and starting the corresponding network service for each of the set number of website files.

[0058] The list of network services can be managed or updated by statistically analyzing the number of received user requests, the forwarding records of user requests, the clients that sent user requests, the paths accessed by user requests, and the startup duration of network services. For example, in Figure 2In the architecture, the generation controller can determine the total frequency of user requests received by each network service in the network service list, the frequency of user requests received from the same client, and the number of user requests currently received by each network service by using the user request reception and forwarding records counted by the receiving and forwarding controller.

[0059] It should be noted that step 1 can be executed before step 2, that is, one or more website files of the first website are generated in advance, and when a user request is received, the website file corresponding to the user request is randomly returned from one or more website files; step 1 can also be executed after step 2, that is, when a user request is received, one or more website files of the first website are generated, or one or more website files corresponding to the user request are generated.

[0060] It should be noted that step 5 can be executed in real time, that is, during the process of providing services on the server side, relevant real-time data is collected to manage or update the network service list in a timely manner.

[0061] To implement the method of the embodiments of this application, the embodiments of this application also provide an anti-crawler device, such as... Figure 3 As shown, the device includes: The generation unit 301 is used to generate one or more website files corresponding to the first website based on the source code of the first website according to the set generation rules. The startup unit 302 is used to start a corresponding network service for each of the one or more website files; wherein the set generation rule is used to modify the location path of the element in the source code; the network service is used to call the corresponding website file to respond to user requests.

[0062] In one embodiment, the setting generation rule includes one or more of the following: Replace the names of the element attributes in the source code with random gibberish; Invalid element tags are randomly inserted into the source code.

[0063] In one embodiment, the device further includes: The receiving unit is used to receive user requests sent by the client; The forwarding unit is used to forward the user request to a network service in the network service list that meets the first set condition, and obtain the corresponding website file; The return unit is used to return the corresponding website file to the client.

[0064] In one embodiment, the first setting condition includes one or more of the following: The total frequency of received user requests is less than or equal to a first set threshold. The startup time is less than or equal to the second set threshold; The number of user requests currently received is less than or equal to the third set threshold; The corresponding weight is the highest, and the weight of network services is determined based on the total frequency of user requests received.

[0065] In one embodiment, the device further includes: An execution unit is configured to perform a first operation when a network service that meets a second preset condition exists in the network service list; the first operation is used to update the network service list.

[0066] In one embodiment, the second setting condition includes one or more of the following: The startup duration is greater than or equal to the fourth set threshold; The total frequency of received user requests is greater than or equal to the fifth set threshold; The frequency of receiving user requests from the same client is greater than or equal to the sixth set threshold.

[0067] In one embodiment, the first operation includes one or more of the following: Stop network services that meet the second set condition; Delete the website files corresponding to the network services that meet the second set conditions; Stop forwarding user requests to network services that meet the second set condition; Determine whether the waiting time of the network service that meets the second set condition is greater than or equal to the seventh set threshold, wherein the waiting time represents the time interval between the last time the network service received the user request and the current time. According to the established generation rules, based on the source code of the first website, a set number of website files corresponding to the first website are regenerated. For each website file in the set number of website files, start the corresponding network service.

[0068] In practical applications, the generation unit 301, the startup unit 302, the forwarding unit, and the execution unit can be implemented by the processor in the anti-crawler device, and the receiving unit and the return unit can be implemented by the processor in the anti-crawler device in combination with the communication interface.

[0069] It should be noted that the anti-crawler device provided in the above embodiments is only illustrated by the division of the above program modules. In practical applications, the above processing can be assigned to different program modules as needed, that is, the internal structure of the device can be divided into different program modules to complete all or part of the processing described above. In addition, the anti-crawler device and the anti-crawler method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0070] Based on the hardware implementation of the above program modules, embodiments of this application also provide an electronic device, such as... Figure 4 As shown, the electronic device 400 includes: Communication interface 401 allows for information exchange with other devices or network nodes.

[0071] The processor 402 is connected to the communication interface 401 to enable information interaction with other devices or network nodes, and to execute the methods provided by one or more technical solutions in the above embodiments when running computer programs.

[0072] Memory 403 is used to store computer programs that can run on processor 402.

[0073] Specifically, the processor 402 is used to generate one or more website files corresponding to the first website based on the source code of the first website according to the set generation rules; For each of the one or more website files, a corresponding network service is started; wherein, the set generation rules are used to modify the location path of elements in the source code; the network service is used to call the corresponding website file to respond to user requests.

[0074] In one embodiment, the setting generation rule includes one or more of the following: Replace the names of the element attributes in the source code with random gibberish; Invalid element tags are randomly inserted into the source code.

[0075] In one embodiment, the communication interface 401 is used to receive user requests sent by a client and return the corresponding website file to the client; The processor 402 is further configured to forward the user request to a network service in the network service list that meets the first set condition, and obtain the corresponding website file.

[0076] In one embodiment, the first setting condition includes one or more of the following: The total frequency of received user requests is less than or equal to a first set threshold. The startup time is less than or equal to the second set threshold; The number of user requests currently received is less than or equal to the third set threshold; The corresponding weight is the highest, and the weight of network services is determined based on the total frequency of user requests received.

[0077] In one embodiment, the processor 402 is further configured to perform a first operation when a network service that meets a second set condition exists in the network service list; the first operation is used to update the network service list.

[0078] In one embodiment, the second setting condition includes one or more of the following: The startup duration is greater than or equal to the fourth set threshold; The total frequency of received user requests is greater than or equal to the fifth set threshold; The frequency of receiving user requests from the same client is greater than or equal to the sixth set threshold.

[0079] In one embodiment, the first operation includes one or more of the following: Stop network services that meet the second set condition; Delete the website files corresponding to the network services that meet the second set conditions; Stop forwarding user requests to network services that meet the second set condition; Determine whether the waiting time of the network service that meets the second set condition is greater than or equal to the seventh set threshold, wherein the waiting time represents the time interval between the last time the network service received the user request and the current time. According to the established generation rules, based on the source code of the first website, a set number of website files corresponding to the first website are regenerated. For each website file in the set number of website files, start the corresponding network service.

[0080] It should be noted that the specific processing procedure of processor 402 can be understood by referring to the above method. Of course, in practical applications, the various components in electronic device 400 are coupled together through bus system 404. It can be understood that bus system 404 is used to realize the connection and communication between these components. In addition to the data bus, bus system 404 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, in... Figure 4 The general designated all buses as Bus System 404.

[0081] The memory 403 in this embodiment is used to store various types of data to support the operation of the electronic device 400. Examples of such data include any computer program used to operate on the electronic device 400.

[0082] The methods disclosed in the embodiments of this application can be applied to the processor 402, or implemented by the processor 402. The processor 402 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit of the hardware in the processor 402 or by instructions in the form of software. The processor 402 may be a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor 402 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the methods disclosed in the embodiments of this application can be directly reflected as being executed by a hardware decoding processor, or being executed by a combination of hardware and software modules in the decoding processor. The software modules may be located in a storage medium, which is located in the memory 403. The processor 402 reads the information in the memory 403 and completes the steps of the aforementioned method in combination with its hardware.

[0083] In an exemplary embodiment, the electronic device 400 may be implemented by one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers (MCUs), microprocessors, or other electronic components to perform the aforementioned method.

[0084] It is understood that the memory (memory 403) in this embodiment of the application can be volatile memory or non-volatile memory, or it can include both volatile memory and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disc, or compact disc read-only memory (CD-ROM); the magnetic surface memory can be disk storage or magnetic tape storage. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Synchronous Static Random Access Memory (SSRAM), Dynamic Random Access Memory (DRAM), Synchronous Dynamic Random Access Memory (SDRAM), Double Data Rate Synchronous Dynamic Random Access Memory (DDRSDRAM), Enhanced Synchronous Dynamic Random Access Memory (ESDRAM), SyncLink Dynamic Random Access Memory (SLDRAM), and Direct Rambus Random Access Memory (DRRAM).The memories described in the embodiments of this application are intended to include, but are not limited to, these and any other suitable types of memories.

[0085] In an exemplary embodiment, this application also provides a storage medium, namely a computer storage medium, specifically a computer-readable storage medium, such as a memory 403 storing a computer program, which can be executed by the processor 402 of the electronic device 400 to complete the steps described in any of the aforementioned methods. The computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, Flash Memory, magnetic surface memory, optical disc, or CD-ROM.

[0086] For example, embodiments of this application also provide a computer program product, including a computer program that can be executed by a processor 402 of an electronic device 400 to perform the steps described in any of the foregoing methods.

[0087] It should be noted that terms such as "first" and "second" are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. The term "and / or" in this document merely describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The terms "one or more" and "one or more" in this document are exemplary expressions and can be replaced with any possible expressions, such as one or more, at least one, or at least one. Furthermore, the technical solutions described in the embodiments of this application can be arbitrarily combined without conflict. The above descriptions are merely preferred embodiments of this application and are not intended to limit the scope of protection of this application.

Claims

1. An anti-crawler method, characterized in that, The method includes: According to the set generation rules, based on the source code of the first website, one or more website files corresponding to the first website are generated, and for each of the one or more website files, a corresponding network service is started; wherein, the set generation rules are used to modify the location path of elements in the source code; the network service is used to call the corresponding website file to respond to user requests.

2. The method according to claim 1, characterized in that, The specified generation rules include one or more of the following: Replace the names of the element attributes in the source code with random gibberish; Invalid element tags are randomly inserted into the source code.

3. The method according to claim 1, characterized in that, The method further includes: Receive user requests sent by the client; The user request is forwarded to a network service in the network service list that meets the first set condition, and the corresponding website file is obtained; The corresponding website file is returned to the client.

4. The method according to claim 3, characterized in that, The first setting condition includes one or more of the following: The total frequency of received user requests is less than or equal to a first set threshold. The startup time is less than or equal to the second set threshold; The number of user requests currently received is less than or equal to the third set threshold; The corresponding weight is the highest, and the weight of network services is determined based on the total frequency of user requests received.

5. The method according to claim 3 or 4, characterized in that, The method further includes: If a network service that meets the second set condition exists in the network service list, a first operation is performed; the first operation is used to update the network service list.

6. The method according to claim 5, characterized in that, The second setting condition includes one or more of the following: The startup duration is greater than or equal to the fourth set threshold; The total frequency of received user requests is greater than or equal to the fifth set threshold; The frequency of receiving user requests from the same client is greater than or equal to the sixth set threshold.

7. The method according to claim 5, characterized in that, The first operation includes one or more of the following: Stop network services that meet the second set condition; Delete the website files corresponding to the network services that meet the second set conditions; Stop forwarding user requests to network services that meet the second set condition; Determine whether the waiting time of the network service that meets the second set condition is greater than or equal to the seventh set threshold, wherein the waiting time represents the time interval between the last time the network service received the user request and the current time. According to the established generation rules, based on the source code of the first website, a set number of website files corresponding to the first website are regenerated. For each website file in the set number of website files, start the corresponding network service.

8. An anti-crawler device, characterized in that, include: The generation unit is used to generate one or more website files corresponding to the first website based on the source code of the first website, according to the set generation rules. The startup unit is used to start a corresponding network service for each of the one or more website files; wherein the set generation rule is used to modify the location path of the element in the source code; the network service is used to call the corresponding website file to respond to user requests.

9. An electronic device, characterized in that, The method includes a processor and a memory for storing a computer program capable of running on the processor, wherein, when the processor is used to run the computer program, it performs the steps of the method according to any one of claims 1 to 7.

10. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

11. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images