Intelligent web crawler method and system

Abstract

The invention provides an intelligent web crawler method and system, and is applied to the technical field of network information processing. The page type of a website is judged through multiple intelligent agents based on a large model, the website is divided into a home page, a list page, a detail page and other pages, the website is compiled into a website map which is of a tree structure and takes the home page as a root, the list page as a path node and the detail page and the other pages as leaf nodes, and when a web crawler collects the website every time, the website map is divided into a plurality of web pages. In the prior art, a map of a current website is compared with a website map acquired in the previous time, and only incremental data is selected for acquisition. By adopting the intelligent web crawler technology provided by the invention, the pressure of the target website is reduced, and the collection efficiency is also improved.

Description

Technical Field

[0001] The present invention relates to the technical field of network information processing, and in particular to an intelligent network crawler method and system. Background Art

[0002] Web page target data scraping is an important means of acquiring and collecting various types of information on the Internet. By scraping target data from web pages, you can obtain a large amount of structured data for data analysis, mining, and research. This data can help companies conduct market research, competitive analysis, and other activities, playing a vital role in decision-making and business optimization. Using web page target data scraping can automate the data collection process, significantly saving manpower and time costs. Compared to manual data scraping, by writing crawler programs, you can automatically scrape large amounts of target data, improving the efficiency and accuracy of data collection.

[0003] Web crawler technology is a method for automatically acquiring internet information. It simulates human behavior, accessing web pages, extracting data, and storing it locally or at other target locations. To improve the quality of accurately crawling target data from web pages, web crawlers are used to crawl target content from various web pages, providing users with reliable data. Traditional web crawlers require a large number of data collection engineers to develop code specifically for target websites. This results in long development cycles and slow response times, which have become urgent technical issues that need to be addressed. Summary of the Invention

[0004] To address the aforementioned issues in the existing technology, the present invention proposes an intelligent web crawler method and system based on a multimodal large language model. Using multiple intelligent agents based on the large model, the system determines the page type of a website and compiles it into a tree-like sitemap with the homepage as the root, list pages as path nodes, and detail pages and other pages as leaf nodes. The web crawler then only selects incremental data for collection, thereby reducing the pressure on the target website while improving collection efficiency.

[0005] The technical solutions adopted in the present invention are as follows:

[0006] An intelligent web crawler method, the method comprising:

[0007] Step S1: Acquire website source data through a web crawler and input it into the website audit agent for audit judgment;

[0008] Step S2: For websites that meet the target scope of the collection, the website navigation agent compiles a website map based on the website source data;

[0009] Step S3: The web crawler performs incremental crawling of the website according to the website map, and stores newly added pages of the website based on the results of the incremental crawling;

[0010] Step S4: The page parsing agent obtains the HTML source code of the newly added page of the website, and parses the page according to the target field. The parsing review agent then reviews the results of the page parsing and stores the results of the page parsing.

[0011] Furthermore, before the above step S1, the method further includes: step S0, providing the collection purpose and target website in natural language, and providing the target field in JSON format.

[0012] Furthermore, the website source data includes the HTML source code of the website homepage, screenshots of the website homepage, and OCR recognition results of the screenshots.

[0013] Furthermore, the website audit agent performs audit judgment, specifically including:

[0014] Step S101: Input the HTML source code of the website homepage in the website source data into the multimodal large language model to determine the website type;

[0015] Step S102: Render the website homepage through the browser and take a screenshot, and simultaneously input the HTML source code of the website homepage into the multimodal large language model to determine whether the website is an abnormal website;

[0016] Step S103: comprehensively judge whether the website meets the collection target range based on the judgment result of the website type and the judgment result of the abnormal website.

[0017] Furthermore, in the above step S102, OCR recognition is performed on the screenshot to determine the actual display content of the website homepage, and the actual display content is compared with the HTML source code of the website homepage. If the comparison result is inconsistent, it indicates that the website homepage has been tampered with, and the website is determined to be an abnormal website; if the comparison result is consistent, the website is a normal website.

[0018] Furthermore, in the above step S2, a website map is compiled based on the website source data, specifically including:

[0019] S201, obtaining the HTML source code and screenshots of the website homepage through a web crawler, and parsing them to obtain OCR recognition results of the screenshots;

[0020] S202, obtaining a content page link included in the website homepage;

[0021] S203: compile a navigation page-list page site map of the target website and store it.

[0022] Furthermore, in the above step S202, obtaining the content page link included in the website homepage specifically includes:

[0023] Determine whether there is a robots.txt file in the root directory of the website. For websites that have a robots.txt file, if the website already has a complete sitemap, directly obtain the existing sitemap;

[0024] For websites that do not have a robots.txt file, or for websites that have a robots.txt file but do not have a complete website map, we will process them based on the properties of the website and obtain all page links on the website's homepage.

[0025] Furthermore, the processing is performed according to the attributes of the website, specifically including:

[0026] If the website is a static website, use the HTML page parsing toolkit to obtain all page links on the website homepage;

[0027] If the website is a dynamic website, the HTML source code of the website homepage is first parsed through a multimodal large language model to obtain the ajax interface address, and the content of the obtained ajax interface address is analyzed one by one to determine all page links in the website homepage.

[0028] Furthermore, in the above step S3, the newly added pages of the website are stored according to the result of the incremental crawling, which specifically includes: comparing the result of the incremental crawling with the stored website map, and storing the website pages that are not in the website map.

[0029] An intelligent web crawler system, which is used to implement the above method, includes a web crawler, a website auditing agent, a website navigation agent, a page parsing agent, a parsing and auditing agent, and a database;

[0030] Web crawlers are used to obtain website source data and input it into the website audit agent for audit judgment. They perform incremental crawling of the website according to the website map and store the newly added pages of the website in the database based on the results of the incremental crawling.

[0031] Website auditing agent, used to conduct audit and judgment based on website source data to determine which websites meet the collection target range;

[0032] Website navigation agent, used to compile website maps for websites that meet the collection target range based on website source data;

[0033] Page parsing agent, used to parse the website page according to the target field entered by the user;

[0034] The parsing and reviewing intelligent agent is used to review the results of page parsing and store the results in the database.

[0035] The following technical effects can be achieved through the embodiments of the present invention:

[0036] (1) Compared with the existing technology, the intelligent web crawler technology adopted by the present invention does not require manual labeling and relies entirely on multi-agent technology to achieve high-performance crawling of target websites;

[0037] (2) The web crawler only selects incremental data for collection each time, which reduces the pressure on the target website and improves the collection efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments or descriptions of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0039] Figure 1 A flowchart of the intelligent network method is shown;

[0040] Figure 2 Schematic diagram of the composition structure of the intelligent network system. DETAILED DESCRIPTION

[0041] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0042] There are two different crawling strategies in web crawling technology: topic crawlers and general crawlers. A topic crawler is a crawler that targets a specific topic or field and can crawl and fetch relevant web pages. A topic crawler determines the scope of its crawl by starting with a seed URL or keyword. Once it finds a relevant page, it will drill down to the link and continue crawling related content. A topic crawler is often used to build search engines, information aggregation websites, and other related fields. A general crawler, also known as a full-web crawler or breadth-first crawler, is a crawler that crawls almost all web pages on the Internet. A general crawler starts with a seed URL and continues to crawl all links in the page to form a breadth-first crawl graph. The goal of a general crawler is to cover as much information as possible on the Internet to support search engines, website indexes, etc.

[0043] Before describing the technical solution of this application, the technical terms used in the technical solution are explained as follows:

[0044] Website content pages: pages containing complete content information, such as news pages, product details pages, recruitment details pages, winning bid details pages, etc.

[0045] Website navigation page: a list page pointing to multiple content pages, such as a specific column in a news website;

[0046] Other pages: pages that are not content pages or navigation pages, such as user login page, about us page, etc.

[0047] On the one hand, the present invention provides an intelligent web crawler method, Figure 1 The figure is a flow chart of an intelligent web crawler method, which includes the following steps:

[0048] Step S1: Acquire website source data through a web crawler and input it into the website audit agent for audit judgment;

[0049] The website source data includes the HTML source code of the website homepage, screenshots of the website homepage, and OCR recognition results of the screenshots;

[0050] The above steps can be considered the website review phase. During this phase, conventional web crawlers are used to collect website source data. The collected data, including the website homepage HTML source code, screenshots of the homepage, and the screenshot OCR results, will be used for review and judgment. If the website meets the collection criteria, the process will jump to the next step. Otherwise, for websites that do not meet the collection criteria, the subsequent processing will be terminated and the process will return directly to the web crawler to obtain source data for other websites.

[0051] Based on the website source data, the website audit agent determines whether the website is within the collection target range and excludes non-news websites (such as yellow pages, recruitment, heterogeneous types, etc.). The website audit agent conducts audit judgment, specifically including:

[0052] Step S101: Input the HTML source code of the website homepage in the website source data into the multimodal large language model to determine the website type;

[0053] Step S102: Render the website homepage through the browser and take a screenshot, and simultaneously input the HTML source code of the website homepage into the multimodal large language model to determine whether the website is an abnormal website;

[0054] In the above step S102, the screenshot is subjected to OCR recognition to determine the actual display content of the website homepage, and the actual display content is compared with the HTML source code of the website homepage. If the comparison result is inconsistent, it indicates that the website homepage has been tampered with and the website is determined to be an abnormal website; if the comparison result is consistent, the website is determined to be a normal website;

[0055] In the above judgment, the content of such abnormal websites is often tampered with through js injection, which is manifested in that the actual displayed content is seriously inconsistent with the HTML source code of the website's homepage. By rendering and taking screenshots through the browser and identifying the screenshots, the actual displayed content of the website's homepage can be accurately obtained, and this can be used as the basis for review and judgment.

[0056] Step S103: comprehensively judge whether the website meets the collection target range based on the judgment result of the website type and the judgment result of the abnormal website.

[0057] Before the above step S1, the method further includes:

[0058] Step S0: Provide the collection purpose and target website in natural language and the target field in JSON format;

[0059] The above steps can be considered as the preliminary preparation stage. Preliminary preparations are carried out based on the collection purpose, target website, and target field.

[0060] To effectively monitor and warn of network information, it is often necessary to incrementally collect content published online, promptly update the published information, process and distribute this content, and then push it to organizations and individuals who need it. The purpose of this collection is to incrementally collect content published online, process the incrementally collected content, and then distribute and push it to users (such as organizations or individuals) who need it.

[0061] There's a huge amount of incremental information online every day, but not all of it is of interest to us. For example, some recruitment websites publish a large amount of information every day, but this isn't what users are interested in. Before collecting data, we need to filter by website type, focusing on news, government, university websites, and news sections from other websites.

[0062] The target website is used to determine the type of website to be collected, and is used as a basis for filtering and screening.

[0063] The target fields include the title, content, release time, author name, picture link, and video link of the news.

[0064] In the above step S1, the result of the audit judgment is stored and regularly updated according to a preset period.

[0065] Storage media typically include memory, disks, or databases. Regular updates ensure that the stored audit results are accurate, timely, and effective. For example, if a legitimate website is hacked and becomes an anomaly, these anomalies can be restored to a legitimate state after repair.

[0066] Step S2: For websites that meet the target scope of the collection, the website navigation agent compiles a website map based on the website source data;

[0067] Different from the website review agent in the above step S1, this website review agent is only responsible for reviewing whether the website meets the collection target range, which is a pre-filtering work. The subsequent website map compilation work is completed by the website navigation agent.

[0068] In the above step S2, the website map is compiled based on the website source data, specifically including:

[0069] S201, obtaining the HTML source code and screenshots of the website homepage through a web crawler, and parsing them to obtain OCR recognition results of the screenshots;

[0070] In the above step S201, the HTML source code and screenshots of the website homepage are newly acquired by the web crawler. In order to reduce system overhead, the HTML source code and screenshots of the website homepage already acquired by the website audit agent and the OCR recognition results of the corresponding screenshots can also be used.

[0071] S202, obtaining a content page link included in the website homepage;

[0072] A website's robots.txt file is a file used by website administrators to instruct web crawlers (such as search engine robots) on how to crawl and index website content. To obtain a website's robots.txt file, simply add the path / robots.txt to the website's root domain name. In step S202 above, obtaining the content page link contained in the website's homepage specifically includes:

[0073] Determine whether there is a robots.txt file in the root directory of the website. For websites that have a robots.txt file, if the website itself already has a complete sitemap, directly obtain the existing sitemap;

[0074] For websites that do not have a robots.txt file, or for websites that have a robots.txt file but do not have a complete sitemap, the website will be processed based on its properties to obtain all page links on the homepage of the website.

[0075] The above mentioned processing is carried out according to the properties of the website, specifically including:

[0076] If the website is a static website, use the HTML page parsing toolkit to obtain all page links on the website homepage;

[0077] The HTML page parsing toolkit includes lxml and beautifulsoup.

[0078] If the website is a dynamic website, the HTML source code of the website homepage is first parsed through the multimodal large language model to obtain the Ajax interface address, and the content of the obtained Ajax interface address is analyzed one by one to determine all page links on the website homepage;

[0079] S203: compile a navigation page-list page site map of the target website and store it.

[0080] In the above step S203, the navigation page-list page site map of the target website is compiled, which specifically includes:

[0081] S2031. Based on all page links and their corresponding link texts obtained from the website homepage, determine the link format, text length, and page category link clustering features of each of the page links, and thereby obtain a probability value of each of the page links belonging to a list page, a detail page, or another page; input the page screenshots, probability values, and page HTML source code corresponding to each of the page links into a multimodal large language model for analysis to determine the type of each link in the page links;

[0082] S2032. For a list page, extract the link content within the page. Input the page screenshot, screenshot OCR, and page HTML source code into a multimodal large language model to obtain a rule for extracting the detail page link within the page. According to the rule, obtain the detail page link within the list page.

[0083] S2033. Compile the website into a tree-like sitemap with the homepage link as the root, the list page link as the intermediate node, and the detail page and other pages as leaf nodes;

[0084] S2034. Store the list page and detail page link extraction rules and the website map in the database respectively.

[0085] The above steps can be considered the sitemap compilation phase. By using a conventional crawler to obtain the HTML source code of the website's homepage, screenshots of the homepage, and the OCR results of the screenshots, the website navigation agent searches for web links that meet the navigation page standards. Using various techniques, the agent then retrieves links to the content pages contained within the navigation page and compiles a navigation-list page sitemap for the target website.

[0086] Step S3: The web crawler performs incremental crawling of the website according to the website map, and stores newly added pages of the website based on the results of the incremental crawling;

[0087] In the above steps, the web crawler crawls the website's site map homepage and list page, and extracts detail page links from the list page according to the detail page extraction rules stored in the database to form the current site map; the web crawler compares the current site map with the site map stored in the database, determines the newly added pages and uses them as incremental data to be crawled.

[0088] In the above step S3, the newly added pages of the website are stored according to the result of the incremental crawling, which specifically includes: comparing the result of the incremental crawling with the stored website map, and storing the website pages that are not in the website map.

[0089] The above comparison process, for example, starts from the root node and traverses all intermediate nodes through the breadth-first method. For the first intermediate node (list page), the details page currently contained therein is compared with the details page contained in the previous version. The newly added content is the newly released content of the website.

[0090] The above steps can be considered as the incremental crawling stage of the website. The website is crawled according to the prepared site map through a web crawler. Pages that are not in the stored site map are identified as new pages and stored, thus achieving incremental crawling of the website.

[0091] Step S4: The page parsing agent obtains the HTML source code of the newly added page of the website, and parses the page according to the target field. The parsing review agent then reviews the results of the page parsing and stores the results of the page parsing.

[0092] In the above steps, page parsing is performed, specifically including: extracting fields from the content within the website according to a preset framework; when extracting fields, if the page format of the website follows a preset page standard (such as ls+json, opengl and other common page standards), field extraction is performed based on the page standard;

[0093] The extracted fields include title, content, release time, and author, and the preset framework includes newspaper3k and readability.

[0094] The extracted fields are input into the multimodal large language model for review according to the preset period. If the review result is consistent with the result of field extraction according to the preset framework or the field extraction following the preset page standard, the field extraction according to the preset framework or the field extraction following the preset page standard is carried out; if not consistent, the preset framework or the preset page standard is modified by the parsing and review intelligent agent, and the modified details page link extraction rule is stored in the database.

[0095] The above steps can be considered the page parsing phase. The page parsing agent generates page parsing code based on the target fields entered by the user. The generated code is then used to parse the page. The parsing review agent and manual review check the page parsing results, which are then stored for subsequent analysis.

[0096] On the other hand, the present invention provides an intelligent web crawler system. Figure 2 This is a schematic diagram of the structure of the intelligent network system. The system includes a web crawler, a website audit agent, a website navigation agent, a page parsing agent, a parsing and auditing agent, and a database.

[0097] Web crawlers are used to obtain website source data and input it into the website audit agent for audit judgment. They perform incremental crawling of the website according to the website map and store the newly added pages of the website in the database based on the results of the incremental crawling.

[0098] Website auditing agent, used to conduct audit and judgment based on website source data to determine which websites meet the collection target range;

[0099] Website navigation agent, used to compile website maps for websites that meet the collection target range based on website source data;

[0100] Page parsing agent, used to parse the website page according to the target field entered by the user;

[0101] The parsing and reviewing intelligent agent is used to review the results of page parsing and store the results in the database.

[0102] In summary, in the technical solution of the present invention, a variety of large-scale model-based intelligent agents are used to determine the page type of a website, dividing the website into homepage, list page, detail page, and other pages. The website is then compiled into a tree-structured website map with the homepage as the root, the list page as the path node, and the detail page and other pages as leaf nodes. Each time the web crawler collects data from a website, it compares the current website map with the previously collected website map and only selects incremental data for collection. This reduces the pressure on the target website and improves collection efficiency.

[0103] Various embodiments of the systems and techniques described above in the present invention can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system including at least one programmable processor, which may be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0104] The program code for implementing the method disclosed in the present invention can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0105] In the context of the present invention, machine-readable medium can be a tangible medium that can contain or store a program for use with an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. Machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. More specific examples of machine-readable storage media can include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0106] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0107] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0108] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact through a communication network. The client-server relationship arises through computer programs running on the respective computers and having a client-server relationship with each other. The server may be a cloud server, a server in a distributed system, or a server integrated with a blockchain.

[0109] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in the present disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in the present disclosure can be achieved. The present disclosure is not limited here.

[0110] The above specific embodiments do not constitute a limitation on the scope of protection disclosed in this invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles disclosed in this invention shall be included within the scope of protection disclosed in this invention.

Claims

1. An intelligent web crawler method, characterized in that: The method includes: Step S1: Acquire website source data through a web crawler and input it into the website audit agent for audit judgment; Step S2: For websites that meet the target scope of the collection, the website navigation agent compiles a website map based on the website source data; Step S3: The web crawler performs incremental crawling of the website according to the website map, and stores newly added pages of the website based on the results of the incremental crawling; Step S4: The page parsing agent obtains the HTML source code of the newly added page of the website, and parses the page according to the target field. The parsing review agent then reviews the results of the page parsing and stores the results of the page parsing.

2. The method according to claim 1, characterized in that Before the above step S1, the method further includes: step S0, providing the collection purpose and target website in natural language, and providing the target field in JSON format.

3. The method according to claim 1 or 2, characterized in that The website source data includes the HTML source code of the website homepage, screenshots of the website homepage, and OCR recognition results of the screenshots.

4. The method according to claim 1, wherein The website audit agent performs audit judgment, specifically including: Step S101: Input the HTML source code of the website homepage in the website source data into the multimodal large language model to determine the website type; Step S102: Render the website homepage through the browser and take a screenshot, and simultaneously input the HTML source code of the website homepage into the multimodal large language model to determine whether the website is an abnormal website; Step S103: comprehensively judge whether the website meets the collection target range based on the judgment result of the website type and the judgment result of the abnormal website.

5. The method according to claim 4, characterized in that In the above step S102, OCR recognition is performed on the screenshot to determine the actual display content of the website homepage, and the actual display content is compared with the HTML source code of the website homepage. If the comparison result is inconsistent, it indicates that the website homepage has been tampered with, and the website is determined to be an abnormal website; if the comparison result is consistent, the website is a normal website.

6. The method according to claim 1, characterized in that In the above step S2, the website map is compiled based on the website source data, specifically including: S201, obtaining the HTML source code and screenshots of the website homepage through a web crawler, and parsing them to obtain OCR recognition results of the screenshots; S202, obtaining a content page link included in the website homepage; S203: compile a navigation page-list page site map of the target website and store it.

7. The method according to claim 6, characterized in that In the above step S202, obtaining the content page link included in the website homepage specifically includes: Determine whether there is a robots.txt file in the root directory of the website. For websites that have a robots.txt file, if the website already has a complete sitemap, directly obtain the existing sitemap; For websites that do not have a robots.txt file, or for websites that have a robots.txt file but do not have a complete website map, we will process them based on the properties of the website and obtain all page links on the website's homepage.

8. The method according to claim 7, characterized in that The above mentioned processing is carried out according to the properties of the website, specifically including: If the website is a static website, use the HTML page parsing toolkit to obtain all page links on the website homepage; If the website is a dynamic website, the HTML source code of the website homepage is first parsed through the multimodal large language model to obtain the ajax interface address, and the content of the obtained ajax interface address is analyzed one by one to determine all page links in the website homepage.

9. The method according to claim 1, characterized in that In the above step S3, the newly added pages of the website are stored according to the result of the incremental crawling, which specifically includes: comparing the result of the incremental crawling with the stored website map, and storing the website pages that are not in the website map.

10. An intelligent web crawler system, the system being used to implement the method according to any one of claims 1 to 9, characterized in that: The system includes a web crawler, a website auditing agent, a website navigation agent, a page parsing agent, a parsing auditing agent, and a database; Web crawlers are used to obtain website source data and input it into the website audit agent for audit judgment. They perform incremental crawling of the website according to the website map and store the newly added pages of the website in the database based on the results of the incremental crawling. Website auditing agent, used to conduct audit and judgment based on website source data to determine which websites meet the collection target range; Website navigation agent, used to compile website maps for websites that meet the collection target range based on website source data; Page parsing agent, used to parse the website page according to the target field entered by the user; The parsing and reviewing intelligent agent is used to review the results of page parsing and store the results in the database.

Images