Webpage information acquisition method and device based on crawler technology, equipment and storage medium

Abstract

The invention relates to a webpage information acquisition method based on a crawler technology, which comprises the following steps of: acquiring a website search interface website, and classifying website structures according to the search interface website: classifying the website structures for the first time according to page number digital information in the search interface website, and classifying the website structures for the second time according to page number digital information in the search interface website; classifying website structures again according to the search keyword information in the search interface website; according to different classifications of the website structures, obtaining a keyword search homepage of the website; judging whether a search entry link of the keyword search home page is hidden or not, and executing storage of the search entry link; and obtaining webpage content linked with the search entry, and judging and screening the webpage content. The invention further relates to a device, equipment and a storage medium.

Description

Technical Field

[0001] The present invention relates to the field of computer application technology, and more specifically to a web page information acquisition method, device, equipment and storage medium based on crawler technology. Background Art

[0002] Obtaining relevant data from websites, such as official websites, is an important part of the work. However, due to the different structures and content forms of websites, some automated crawler technologies have been applied to data collection in related technologies, but in the face of the complexity and variability of different websites, there are many deficiencies in dealing with differences in website structure and responding to anti-crawler measures. Summary of the invention

[0003] To address the above-mentioned shortcomings, according to a first aspect of the present invention, a method for acquiring web page information based on crawler technology is provided, comprising: acquiring the URL of a website search interface, and classifying the website structure according to the search interface URL, including: initially classifying the website structure according to page number information in the search interface URL, and classifying the website structure again according to search keyword information in the search interface URL; acquiring the keyword search homepage of the website according to different classifications of the website structure; determining whether the search item link of the keyword search homepage is hidden, and saving the search item link; and acquiring the web page content of the search item link, and determining and filtering the web page content.

[0004] Optionally, the initial classification of the website structure includes: a first category: the search interface URL contains page numbers; and a second category: the search interface URL does not contain page numbers.

[0005] Optionally, if the website structure is initially classified as the first category, the method further includes: determining the page jump type.

[0006] Optionally, the page number jump type includes five different types.

[0007] Optionally, the website structure is further classified into: a third category: the search interface URL contains search keywords; and a fourth category: the search interface URL does not contain search keywords.

[0008] Optionally, if the website structure is classified into the fourth category again, the method further includes: simulating manual input of keyword search.

[0009] Optionally, the determining whether the search item link of the keyword search homepage is hidden and executing the saving of the search item link, if the search item link is hidden, further comprises: simulating clicking the search item to obtain dynamic link content.

[0010] Optionally, the method further includes: detecting and responding to anti-crawler information of the website to obtain web page content.

[0011] The step of obtaining the webpage content linked to the search item, and judging and screening the webpage content, further includes: setting a scoring function to evaluate the matching degree between the webpage content and the expected target, satisfying the following formula:

[0012]

[0013] Among them, I i (text) is the indicator function, ω i It is the weight of the regular expression that describes the search keyword.

[0014] According to a second aspect of the present invention, there is provided a web page information acquisition device based on crawler technology, comprising: a classification module: used to acquire the URL of a website search interface, and classify the website structure according to the search interface URL, including: initially classifying the website structure according to page number information in the search interface URL, and classifying the website structure again according to search keyword information in the search interface URL; an acquisition module: used to acquire the keyword search homepage of the website according to different classifications of the website structure; a saving module: used to determine whether the search item link of the keyword search homepage is hidden, and execute the saving of the search item link; a screening module: used to acquire the web page content of the search item link, and determine and screen the web page content.

[0015] According to a third aspect of the present invention, there is provided an electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the above-mentioned method.

[0016] According to a fourth aspect of the present invention, there is provided a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause the computer to execute the above method.

[0017] According to a fifth aspect of the present invention, there is provided a computer program product, comprising a computer program, wherein the computer program implements the steps of the above method when executed by a processor.

[0018] The embodiments of the present invention can improve the efficiency and accuracy of web page information acquisition and reduce manual operation time. Through automated crawler technology, different website structures and anti-crawler mechanisms can be effectively dealt with, web page content can be screened and verified, and the validity and reliability of information can be ensured. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The following are drawings of exemplary embodiments of the present invention, and the same or similar reference numerals are used in the drawings to represent the same or similar elements. In the drawings:

[0020] Figure 1 A flow chart of a method for acquiring web page information based on crawler technology according to an exemplary embodiment of the present invention is shown.

[0021] Figure 2 A schematic structural diagram of a web page information acquisition device based on crawler technology according to an exemplary embodiment of the present invention is shown.

[0022] Figure 3 A schematic structural diagram of an electronic device according to an exemplary embodiment of the present invention is shown. DETAILED DESCRIPTION

[0023] In order to better explain the present invention and facilitate understanding, the present invention is described in detail below through specific implementation modes in conjunction with the accompanying drawings.

[0024] In the present invention, the term "and / or" is intended to cover all possible combinations and sub-combinations of the listed elements, including any one, any sub-combination or all of the elements listed separately, without excluding other elements. Unless otherwise specified, the terms "first", "second", etc. are used to describe various elements without intending to limit the positional relationship, timing relationship or importance relationship of these elements. Such terms are only used to distinguish one element from another. Unless otherwise specified, the orientation or positional relationship indicated by the terms "front, back, up, down, left, right", etc. is usually based on the orientation or positional relationship shown in the accompanying drawings, which is only for the convenience and simplification of description, and cannot be understood as limiting the scope of protection of the present invention.

[0025] At present, when dealing with differences in the structure of different websites, web crawler technology often requires writing crawler code for each website separately, which has high maintenance costs and low efficiency. At the same time, many official websites have set up anti-crawler mechanisms, which increases the difficulty of data collection. In addition, even if a large amount of data is successfully obtained, how to efficiently filter out useful information also needs further optimization.

[0026] Figure 1 A flow chart of a method for acquiring web page information based on crawler technology according to an exemplary embodiment of the present invention is shown.

[0027] S102: Obtain the website search interface URL, and classify the website structure according to the search interface URL, including: initially classifying the website structure according to page number information in the search interface URL, and reclassifying the website structure according to search keyword information in the search interface URL.

[0028] Using web crawler technology, we conduct structural analysis on each city's official website to identify different web page layouts and content formats. The system needs to classify different websites into structural categories:

[0029] First, the URLs of the website search interface are divided into two categories. For example, the first category: URLs containing page numbers; the second category: URLs without page numbers.

[0030] In some embodiments, for URLs containing page numbers, it is also necessary to determine the page jump type. Further, the page jump types are divided into 5 categories: 1-[p,c], 2-[c,p], 3-[p,c,c], 4-[c,p,c], 5-[c,c,p]. Among them, p represents the current page number, and c represents the content of the search keyword. For example: the following address "http: / / search.gd.gov.cn / search / all / 200001?page={}&keywords={}" involves "page" information and "keywords" information successively, which belongs to the page jump type "1-[p,c]".

[0031] For URLs that do not contain page numbers (the second category), determine whether the URL contains information about search keywords. If it does not contain search keyword content, use the Selenium program to simulate the website's manual keyword search function. Specifically, first, find the structural position of the search box on the website, such as "https: / / www.zhengzhou.gov.cn / ", the structural position of the search box on this page is " / / input[@name='q']", and the structural position of the search button is " / / input[@aria-label='search']". After determining the search box and search button structural positions, use Selenium to input keywords to simulate manual search functions.

[0032] In some other embodiments, according to the website search page URL W i Different websites can be divided into three different types of websites. The first type of website has a page number p in its URL, which belongs to Model 1 The second type of website URL does not contain page numbers but contains search keywords, which belongs to Model 2 The third type of website has a URL that contains neither page numbers nor search keywords, and belongs to Model 3 Structure. As shown in formula (1):

[0033]

[0034] S104: Obtaining a keyword search homepage of the website according to different categories of the website structure.

[0035] S106: Determine whether the search item link of the keyword search homepage is hidden, and save the search item link.

[0036] After different website addresses are classified, the above method is used to obtain the web page of the keyword search homepage. In some embodiments, different search pages are browsed in a loop according to the number of search pages specified by the user. For example, if the user wants to search 20 pages of content, the system will loop 20 times to collect the content of 20 pages of search web pages. After obtaining the keyword search homepage of the website, it is determined whether the URL of each current search item link is hidden.

[0037] If it is not hidden, determine the location structure of the link, such as the a field, url field, data-url field, etc., and then use the regular expression:

[0038] (r'.*gov.cn / .*(.shtml|.html|.htm|content.*|id.*|[0-9]+.aspx|.asp\?id.*|.jsp\?jcnmlgb.*|index\?code=[0-9]+|action\?messagekey=[0-9]+).*') Determine whether the URL meets the expected requirements.

[0039] If the link is hidden, it means that the link is a dynamic link. First locate the website location of the search item link, such as "div.listBox a", and then use Selenium to simulate clicking each search item, and open the website corresponding to the search item link in a new browser window to obtain the specific information of the search link.

[0040] After obtaining the content of all search item links on the current page, in order to crawl the specified number of browsing pages, it is necessary to locate the website structure of the page turning button, such as " / / a[text()='next page']", simulate the click and turn the page by using the Selenium program, and then loop again to obtain the address of the search item link with the above method until the specified number of search pages is met.

[0041] S108: Obtain the webpage content linked to the search item, and judge and filter the webpage content.

[0042] Regular expressions, natural language processing (NLP) technology and machine learning algorithms are used to analyze the acquired web page content and filter out valid information pages. A comprehensive judgment is made based on the frequency of keyword occurrence, page structure and text content to improve the accuracy of the screening results. Specifically, all the search item links crawled above are traversed, each search item link is opened using the Selenium program, and then the text content (i.e., text structure) of the web page is copied.

[0043] Furthermore, first, the copied text content is preprocessed to remove some of the content containing stop words (such as "website", "contact number", "repealed", etc.). Then, a machine learning algorithm is used to determine the validity of the text content. According to a series of regular expressions: R 1 ,R 2 ,...,R n , for example, "(?=.*Economy.*)(?=.*Green.*|.*Low-carbon.*|.*Circular.*|.*Carbon Peak.*|.*Carbon Neutral.*)(?=.*Finance.*|.*Financing.*)(?=.*System.*|.*Notice.*|.*Policy.*|.*Scheme.*)(?!.*Tender.*)(?!.*Winning.*)(?!.*Invalidation.*)(?!.*Abolition.*)", this expression can filter out text content related to green economy, carbon peak and carbon neutrality. Each regular expression is used to match a specific pattern in the text. In addition, each regular expression has an associated weight ω i , indicating its importance in the overall verification process. In addition, a scoring function S is set to calculate the matching degree between the web page text content and the expected target:

[0044]

[0045] Among them, I i (text) is an indicator function. If the webpage text content text matches the regular expression R i , then it returns 1, otherwise it returns 0. i is the regular expression R i The weight of indicates its importance in the overall validity judgment. When S(text) exceeds a certain threshold T, the text content of the webpage meets the requirements and is judged to be valid, otherwise it will continue to jump to the next website link.

[0046] In some embodiments, the anti-crawler information of the website is detected and responded to by simulating manual operation, using proxy IP, dynamically adjusting access frequency, etc., to obtain web page content. The type of anti-crawler measures is automatically detected and corresponding response strategies are adopted. For example, each time a web page is opened, it will stay for 3-5 seconds to simulate the speed of a person browsing the web page. In addition, when setting the browser, the system will add some restrictions to ensure that the operation of the browser is similar to that of a real person. For another example, adding the browser setting code can make the browser run more naturally so that it can pass the website's anti-crawler test.

[0047] In addition, a function of dynamically adjusting the access frequency is also set to prevent the website from blocking the IP due to a large number of visits to the website in a short period of time. As shown in the following formula (3), the access frequency is determined by the parameters k, λ and the time interval t. When the time interval between web page accesses increases (such as large network delays that slow down web page access), the access frequency will increase, and when the time interval decreases, the access frequency will be reduced to prevent too frequent access.

[0048] Access frequency = k·e λt ; (3)

[0049] Finally, the filtered valid web page content is saved as a PDF file for subsequent viewing and use. The system uses Python program to complete the generation and storage of PDF.

[0050] In some embodiments, taking "https: / / www.hangzhou.gov.cn / " as an example, the search term "low carbon" is entered in the search interface to search, and the webpage address of the keyword search homepage of the website is obtained as "https: / / search.zj.gov.cn / jrobotfront / search.do?q=%E4%BD%8E%E7%A2%B3&websiteid=330101000000000&tpl=2296&p=1&municipal_webid=330101000000000&municipal_name=%E6%9D%AD%E5%B7%9E%E5%B8%82&isCo ntains=0&siteCode=330101000000000&”, it can be seen that the above link address contains the search keyword (q=%E4%BD%8E%E7%A2%B3) and the page number (p=1), and the jump type can be judged as “2-[c,p]”. Next, check whether the link of the current web page search item is hidden. The link of the above web page is a static link and does not need to be dynamically obtained. The URL link can be saved directly. Check the structure of the “Next Page” jump button in the web page, which is “ / / li[text()='>”]”, and traverse the next page search page until the specified number of search pages is completed and the loop ends. Furthermore, when using Selenium to initialize the browser, adding the following code can enable the browser to simulate real people clicking to browse the web page:

[0051] option.add_experimental_option('excludeSwitches',['enable-automation'])

[0052] option.add_experimental_option('useAutomationExtension',False)

[0053] option.add_argument("--window-size=1920,1080")

[0054] option.add_argument("--disable-extensions")

[0055] option.add_argument("--headless")

[0056] option.add_argument("--disable-gpu")

[0057] option.add_argument("--disable-software-rasterizer")

[0058] option.add_argument('--no-sandbox')

[0059] option.add_argument('--ignore-certificate-errors')

[0060] option.add_argument('--allow-running-insecure-content')

[0061] option.add_argument("blink-settings=imagesEnabled=false")

[0062] In addition, the system sets the optimal values ​​of parameters k and λ according to the actual crawler effect, so that the time interval when the browser switches web pages can be maintained at about 3 to 5 seconds, which is similar to the time of real manual web browsing.

[0063] Traverse the saved links, open each linked web page, and use machine learning algorithms to determine the relevance of the web page text content, such as multiple regular expressions related to low-carbon environmental protection: (?=.*Green.*|.*Low-carbon.*|.*Circular.*|.*Carbon peak.*|.*Carbon neutral.*); (?=.*Laws.*|.*Regulations.*|.*Specifications.*|.*Methods.*|.*Schemes.*|.*Agreements.*); (?=.*Implementation.*|.*Guarantees.*), etc. Then calculate S(text) of each web page according to formula (3) and compare it with the threshold T. If it is greater than T, the web page content is valid content, otherwise the next web page link is judged. Finally, save the content.

[0064] According to an embodiment of the present invention, Figure 2 is a schematic diagram of a framework of a web page information acquisition device based on crawler technology according to an exemplary embodiment of the present invention. Figure 2 As shown, an embodiment of the present invention further provides a web page information acquisition device based on crawler technology, and the web page information acquisition device 200 based on crawler technology includes:

[0065] Classification module 202: used to obtain the website search interface URL, and classify the website structure according to the search interface URL, including: first classifying the website structure according to the page number information in the search interface URL, and reclassifying the website structure according to the search keyword information in the search interface URL;

[0066] Acquisition module 204: used to acquire the keyword search homepage of the website according to different categories of the website structure;

[0067] Saving module 206: used to determine whether the search item link of the keyword search homepage is hidden, and save the search item link;

[0068] The screening module 208 is used to obtain the webpage content linked by the search item, and to judge and screen the webpage content. It should be noted that the device embodiment in the embodiment of the present invention can refer to the above method embodiment, which will not be described in detail here.

[0069] According to an embodiment of the present invention, the present invention also provides an electronic device, a readable storage medium and a computer program product.

[0070] According to an embodiment of the present invention, the present invention provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable the computer to execute any one of the above methods.

[0071] According to an embodiment of the present invention, the present invention provides a computer program product, which includes: a computer program, the computer program is stored in a readable storage medium, at least one processor of an electronic device can read the computer program from the readable storage medium, and at least one processor executes the computer program so that the electronic device executes the solution provided by any of the above embodiments.

[0072] According to an embodiment of the present invention, the present invention further provides an electronic device, Figure 3 A schematic block diagram of an example electronic device 400 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0073] like Figure 3As shown, the device 400 includes a computing unit 401, which can perform various appropriate actions and processes according to a computer program stored in a ROM 402 or a computer program loaded from a storage unit into a RAM 403. In the RAM 403, various programs and data required for the operation of the device 400 can also be stored. The computing unit 401, the ROM 402, and the RAM 403 are connected to each other via a bus 404. An I / O interface 405 is also connected to the bus 404.

[0074] A number of components in the device 400 are connected to the I / O interface 405, including: an input unit 406, such as a keyboard, a mouse, etc.; an output unit 407, such as various types of displays, speakers, etc.; a storage unit 408, such as a disk, an optical disk, etc.; and a communication unit 409, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 409 allows the device 400 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0075] The computing unit 401 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 401 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 401 performs the various methods and processes described above, such as the problem handling method. For example, in some embodiments, the problem handling method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 408. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 400 via ROM 402 and / or communication unit 809. When the computer program is loaded into RAM 403 and executed by the computing unit 401, one or more steps of the problem handling method described above may be performed. Alternatively, in other embodiments, the computing unit 401 may be configured to perform the problem handling method in any other appropriate manner (e.g., by means of firmware).

[0076] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0077] The program code for implementing the method of the present invention can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code can be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.

[0078] In the context of the present invention, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0079] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube)) or an LCD for displaying information to the user; and a keyboard and pointing device through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback; and input from the user can be received in any form.

[0080] The systems and techniques described herein can be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components, or a computing system that includes frontend components, or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected by digital data communication (e.g., a communication network) in any form or medium. Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0081] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or a cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services. The server may also be a server of a distributed system, or a server combined with a blockchain.

[0082] The embodiments of the present invention can improve the efficiency and accuracy of web page information acquisition and reduce manual operation time. Through automated crawler technology, different website structures and anti-crawler mechanisms can be effectively dealt with, and web page content can be screened and verified to ensure the validity and reliability of information. The method of the embodiments of the present invention is highly versatile and applicable to websites of different structures and formats, has a wide range of application prospects, and has a flexible system architecture, which can be expanded and upgraded according to actual needs.

[0083] It is to be understood that the present invention is described by some embodiments, and it is known to those skilled in the art that various changes or equivalent substitutions may be made to these features and embodiments without departing from the spirit and scope of the present invention. In addition, under the teachings of the present invention, these features and embodiments may be modified to adapt to specific circumstances and materials without departing from the spirit and scope of the present invention. Therefore, the present invention is not limited by the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are within the scope of protection of the present invention.

Claims

1. A web page information acquisition method based on crawler technology, characterized in that: include: Obtaining a website search interface URL, and classifying the website structure according to the search interface URL, including: Based on the page number information in the URL of the search interface, the website structure is initially classified. Reclassify the website structure according to the search keyword information in the search interface URL; According to different categories of the website structure, obtaining the keyword search homepage of the website; Determining whether the search item link of the keyword search homepage is hidden, and saving the search item link; and The webpage content linked to the search item is obtained, and the webpage content is judged and filtered.

2. The web page information acquisition method based on crawler technology as claimed in claim 1, characterized in that: The initial classification of the website structure includes: Category 1: the search interface URL contains page numbers; and Category 2: The search interface URL does not contain page numbers.

3. The web page information acquisition method based on crawler technology as claimed in claim 2, characterized in that: If the website structure is initially classified as the first category, the method further includes: determining the page jump type.

4. The web page information acquisition method based on crawler technology as claimed in claim 3, characterized in that: The page jump type includes five different types.

5. The web page information acquisition method based on crawler technology as claimed in claim 4, characterized in that: The website structure is further classified into: Category 3: The search interface URL contains search keywords; and Category 4: The search interface URL does not contain search keywords.

6. The webpage information acquisition method based on crawler technology as claimed in claim 5, characterized in that: If the website structure is classified into the fourth category again, the method further includes: simulating manual keyword input search.

7. The webpage information acquisition method based on crawler technology as claimed in claim 6, characterized in that: The determining whether the search item link of the keyword search homepage is hidden and executing the saving of the search item link, if the search item link is hidden, further comprises: simulating clicking the search item to obtain dynamic link content.

8. The webpage information acquisition method based on crawler technology as claimed in claim 7, characterized in that: The step of obtaining the webpage content linked to the search item, and judging and screening the webpage content, further includes: setting a scoring function to evaluate the matching degree between the webpage content and the expected target, satisfying the following formula: Among them, I i (text) is the indicator function, ω i It is the weight of the regular expression that describes the search keyword.

9. The webpage information acquisition method based on crawler technology as claimed in claim 1, characterized in that: Also includes: Detect and respond to anti-crawler information of the website to obtain web page content.

10. A web page information acquisition device based on crawler technology, characterized in that: include: Classification module: used to obtain the website search interface URL, and classify the website structure according to the search interface URL, including: first classifying the website structure according to the page number information in the search interface URL, and reclassifying the website structure according to the search keyword information in the search interface URL; Acquisition module: used to acquire the keyword search homepage of the website according to different categories of the website structure; Saving module: used to determine whether the search item link of the keyword search homepage is hidden, and save the search item link; Screening module: used to obtain the webpage content linked to the search item, and to judge and screen the webpage content.

11. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, wherein the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claim 19.

12. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 19.

13. A computer program product, comprising a computer program, which, when executed by a processor, implements the steps of the method according to any one of claims 19.

Images