Web crawler template determination method and device based on large model

Abstract

The invention belongs to the technical field of artificial intelligence, and particularly relates to a web crawler template determination method and device based on a large model, and the method comprises the steps: extracting a network entry page from a to-be-accessed link list, accessing the network entry page, and extracting link related information of the network entry page, the link related information comprises a navigation link and a related website link; adding the navigation link and the related website link to a to-be-accessed link list; traversing a navigation link and a related website link in the to-be-accessed link list to obtain a target sub-page; and analyzing the target sub-page by using the large model to obtain a crawler template. According to the technical scheme provided by the invention, the network page maintenance cost is reduced, the consistency and standardization level of data processing are improved, the automation level and adaptability of data acquisition are improved, the resource use efficiency is remarkably optimized, and a new path is opened up for data acquisition and analysis in the big data era.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of artificial intelligence, and particularly relates to a network crawler template determination method and device based on a large model. BACKGROUND

[0002] In today's rapidly developing network information technology, network crawlers, as one of the core technologies of data collection and analysis in the big data era, play a crucial role. With the exponential growth of Internet content, manually collecting the required information becomes neither realistic nor economical, while network crawlers fill this gap with automation and high efficiency. They can continuously traverse the global network, systematically scraping massive data from web pages, social media, forums, e-commerce platforms, and other online resources, providing a solid foundation for data-driven decision-making in various industries.

[0003] Especially in the field of artificial intelligence, network crawler technology plays a data fueling role in the training of large models for deep learning and machine learning. Large models, such as language models and image recognition models, highly depend on a large amount of diverse training data for their performance. Network crawlers, by specifically crawling text, images, videos, and other content on the Internet, construct a large training dataset, enabling the model to learn more rich and complex data patterns, thereby improving prediction accuracy and generalization ability. For example, in natural language processing tasks, crawlers can scrape social media conversations, news articles, academic papers, and other texts to provide real-time updated, widely context-covered corpora for model training.

[0004] In the daily operation of network crawlers, one challenge that cannot be ignored is the frequent modification of target websites. The Internet world is constantly changing, and many websites will adjust their interfaces and refactor their codes from time to time to improve user experience, add functions, or for security considerations. This is undoubtedly a thorny problem for network crawlers that rely on specific web page structures for data scraping. When the web page structure changes, the well-designed crawler template may instantly fail, causing the crawling task to be interrupted and data scraping errors to occur frequently. At this time, developers need to intervene in time to analyze the new structure of the web page and update the parsing rules and data extraction logic of the crawler accordingly. This process not only requires developers to have the ability to quickly understand and adapt to the new web page layout, but also requires a deep understanding of the crawler framework to efficiently adjust the code, sometimes even rewriting part of the modules. Manual maintenance of this process not only consumes time and effort, but also may affect the continuity and timeliness of data collection, especially when facing large-scale crawler systems and multiple website modifications simultaneously, the maintenance cost rises sharply. SUMMARY

[0005] To overcome the problems in the above related technologies, the application provides a large model-based network crawler template determination method and device.

[0006] According to a first aspect of the embodiments of the application, a large model-based network crawler template determination method is provided, comprising:

[0007] extracting a network entry page from a to-be-accessed link list, accessing the network entry page and extracting link-related information of the network entry page, the link-related information comprising: navigation links and related website links;

[0008] adding the navigation links and the related website links to the to-be-accessed link list;

[0009] traversing the navigation links and the related website links in the to-be-accessed link list to obtain target subpages;

[0010] analyzing the target subpages by using a large model to obtain a crawler template.

[0011] Preferably, the link-related information further comprises:

[0012] a navigation bar name, a navigation link corresponding url, and a related website link corresponding url.

[0013] Preferably, the extracting of the link-related information of the network entry page in the to-be-accessed link list comprises:

[0014] extracting the page id and the link-related information of the network entry page by using a large model;

[0015] storing the page id and the link-related information of the network entry page to a database.

[0016] Preferably, the method further comprises:

[0017] after accessing the network entry page, obtaining a script file of the network entry page;

[0018] storing the script file of the network entry page to a database.

[0019] Preferably, the method further comprises:

[0020] after extracting the link-related information of the network entry page, analyzing the network entry page by using a large model to obtain a page style and a page style id of the network entry page;

[0021] storing the page style and the page style id of the network entry page to a database.

[0022] Preferably, the traversing the navigation links and the related website links in the to-be-accessed link list to obtain the target sub-page comprises:

[0023] The traversing the navigation links and the related website links in the to-be-accessed link list to obtain the initial sub-page corresponding to the navigation links and the related website links of the network entry page;

[0024] The removing the duplicate pages in the initial sub-page to obtain the target sub-page.

[0025] Preferably, the analyzing the target sub-page by using a large model to obtain a crawler template comprises:

[0026] Accessing the target sub-page, judging whether the page style of the target sub-page is same as the page style of the already accessed page, if yes, no need to judge the page type of the target sub-page, if no, determining the page type of the target sub-page by using a large model to obtain the page type of the target sub-page;

[0027] Based on the page type of the target sub-page, analyzing the selector of the target sub-page by using a large model;

[0028] Constructing the crawler template by using the selector of the target sub-page.

[0029] Preferably, the determining the page type of the target sub-page by using a large model comprises:

[0030] When the target sub-page includes first information, the page type of the target sub-page is a detail page;

[0031] When the target sub-page includes second information, the page type of the target sub-page is a list page.

[0032] Preferably, the first information at least includes the following six kinds: title, text content, publication date, source information, access volume and responsible editor information;

[0033] The second information at least includes the following three kinds: list item, list item title and link to other detailed content summary.

[0034] Preferably, the analyzing the selector of the target sub-page by using a large model based on the page type of the target sub-page comprises:

[0035] If the page type of the target sub-page is a list page, a large model is used to analyze the selector of the target sub-page to obtain a list item name selector, a list item link selector, a page turning operation, a page style, and a page style id, and the list item name selector, the list item link selector, the page turning operation, the page style, and the page style id are stored in a database;

[0036] If the page type of the target sub-page is a detail page, a large model is used to analyze the selector of the target sub-page to obtain a text html selector and a title html selector of the target sub-page, and the text html selector and the title html selector are stored in a database.

[0037] Preferably, the method further comprises:

[0038] After accessing the target sub-page, a script file of the target sub-page is obtained;

[0039] The script file of the target sub-page is stored in a database;

[0040] A page id and link related information of the target sub-page are extracted by using a large model, and are stored in a database;

[0041] Navigation links and related website links in the link related information of the target sub-page are added to the list of links to be accessed.

[0042] Preferably, the method further comprises:

[0043] The crawler template is updated regularly.

[0044] Preferably, the crawler template is updated regularly, comprising:

[0045] The title html selector and the text html selector of all detail pages in the database are accessed by using a crawler technology, and when a detail page is entered from a list page, the title html selector and the text html selector of the detail page corresponding to each list item in the list page are accessed, and the page is turned according to the page turning operation of the list page;

[0046] When the title html selector and / or the text html selector of the accessed detail page are invalid, a large model is used to rejudge whether the page style of the detail page is the same as the page style of the page that has been accessed, if the same, the selector of the detail page does not need to be analyzed, if not the same, the selector of the detail page is analyzed by using a large model to obtain the text html selector and the title html selector of the detail page, and the text html selector and the title html selector of the detail page are used to update the crawler template;

[0047] The page id corresponding to the detail page in the accessed database, the linked url, the list item name of the list page, the text, the page state, and the storage time record time stored to the database are stored to the database.

[0048] Preferably, the periodically updating the crawler template further comprises:

[0049] When there is an invalid selector of a target sub-page, a large model is used to determine the page type of the target sub-page corresponding to the invalid selector, the selector corresponding to the target sub-page corresponding to the invalid selector is obtained, and the crawler template is updated using the target sub-page corresponding to the invalid selector; or

[0050] The network entry page in the database is periodically accessed using a crawler technology, the navigation link and the related website link of the network entry page are traversed, when there is a new target sub-page, a large model is used to determine the page type of the new target sub-page, the selector corresponding to the new target sub-page is obtained, and the crawler template is updated using the selector corresponding to the new target sub-page.

[0051] Preferably, the method further comprises: storing a crawling content table using a database;

[0052] The crawling content table comprises: a page id corresponding to a detail page, a url of a link corresponding to the detail page, a list item name of a list page corresponding to the detail page, a text of the detail page, a page state of the detail page, and a storage time record time of the detail page stored to the database.

[0053] Preferably, the method further comprises:

[0054] When a page is accessed, if the page returns 404, the access to the page is stopped, the page state of the page is invalid, and the page state of the page is stored to the database; if the page returns error information, the page is re-accessed at a preset time interval, and it is determined whether the number of times of re-accessing the page reaches a preset number of times; if the page does not return 404 and error information, the page state of the page is normal, and the page state of the page is stored to the database;

[0055] If the number of times of re-accessing the page reaches the preset number of times, the access to the page is stopped, the page state of the page is invalid, and the page state of the page is stored to the database; if the number of times of re-accessing the page does not reach the preset number of times, the page is re-accessed at a preset time interval;

[0056] The accessed page comprises the network entry page or the target sub-page.

[0057] Preferably, the method further comprises:

[0058] storing the page style table, the link relationship table of each page and the storage location of the script file of each page in the database;

[0059] The each page includes the network entry page or the target sub-page.

[0060] Preferably, the page style table includes: each type of page style, the id of each type of page style, the storage path of each page corresponding to each type of page style in the database, the url of each type of page style first stored, the storage time of each type of page style stored to the database, the title html selector of the detail page, the body html selector of the detail page, the list item name selector of the list page, the list item link selector of the list page, the page turning operation of the list page, and whether each page corresponding to the page style is a detail page.

[0061] The link relationship table of each page includes: the page id of each page, the url of the navigation link and the related website link of each page, the parent link id of the navigation link and the related website link of each page, the page style id corresponding to each page, whether the detail page is unique, the storage time of each page stored to the database, and the page state of each page.

[0062] The detail page is unique if the detail page is not entered from the list page, otherwise, the detail page is not unique.

[0063] According to a second aspect of the embodiments of the present application, a network crawler template determination device based on a large model is provided, comprising:

[0064] An extraction unit is configured to extract a network entry page from a to-be-visited link list, visit the network entry page and extract link-related information of the network entry page, the link-related information including navigation links and related website links.

[0065] An adding unit is configured to add the navigation links and the related website links to the to-be-visited link list.

[0066] A first obtaining unit is configured to traverse the navigation links and the related website links in the to-be-visited link list to obtain a target sub-page.

[0067] A second obtaining unit is configured to analyze the target sub-page by using a large model to obtain a crawler template.

[0068] Preferably, the link-related information further includes:

[0069] The navigation bar name, the url corresponding to the navigation link, and the url corresponding to the related website link.

[0070] Preferably, the extraction unit comprises:

[0071] The first extraction module is configured to extract the page id and the link-related information of the network entry page by using the large model.

[0072] The first storage module is configured to store the page id and the link-related information of the network entry page in the database.

[0073] Preferably, the first extraction module is further configured to:

[0074] After accessing the network entry page, the script file of the network entry page is obtained.

[0075] The script file of the network entry page is stored in the database.

[0076] Preferably, the extraction unit further comprises:

[0077] The first analysis module is configured to analyze the network entry page by using the large model to obtain the page style and the page style id of the network entry page after extracting the link-related information of the network entry page.

[0078] The second storage module is configured to store the page style and the page style id of the network entry page in the database.

[0079] Preferably, the first acquisition unit comprises:

[0080] The first acquisition module is configured to traverse the navigation link and the related website link in the to-be-accessed link list to obtain the initial sub-page corresponding to the navigation link and the related website link of the network entry page.

[0081] The second acquisition module is configured to eliminate the repeated pages in the initial sub-page to obtain the target sub-page.

[0082] Preferably, the second acquisition unit comprises:

[0083] The judgment module is configured to access the target sub-page, judge whether the page style of the target sub-page is the same as the page style of the already-accessed page, if the same, the page type of the target sub-page does not need to be judged, and if not the same, the page type of the target sub-page is determined by using the large model to obtain the page type of the target sub-page.

[0084] The second analysis module is configured to analyze the selector of the target sub-page by using the large model based on the page type of the target sub-page.

[0085] a constructing module, configured to construct the crawler template by using the selector of the target sub-page.

[0086] Preferably, the judging module is further configured to:

[0087] when the target sub-page includes the first information, the page type of the target sub-page is a detail page;

[0088] when the target sub-page includes the second information, the page type of the target sub-page is a list page.

[0089] Preferably, the first information includes at least the following six types: title, text content, publication date, source information, access volume and responsible editor information;

[0090] the second information includes at least the following three types: list item, list item title and link to other detailed content.

[0091] Preferably, the second analyzing module is specifically configured to:

[0092] if the page type of the target sub-page is a list page, analyze the selector of the target sub-page by using a large model to obtain a list item name selector, a list item link selector, a page turning operation, a page style and a page style id, and store the list item name selector, the list item link selector, the page turning operation, the page style and the page style id into a database;

[0093] if the page type of the target sub-page is a detail page, analyze the selector of the target sub-page by using a large model to obtain a body html selector and a title html selector of the target sub-page, and store the body html selector and the title html selector into a database.

[0094] Preferably, the judging module is further configured to:

[0095] after accessing the target sub-page, obtain a script file of the target sub-page;

[0096] store the script file of the target sub-page into a database;

[0097] extract a page id and link related information of the target sub-page by using a large model, and store them into a database;

[0098] add navigation links and related website links in the link related information of the target sub-page to the list of links to be accessed.

[0099] Preferably, the device further comprises:

[0100] an updating unit configured to periodically update the crawler template.

[0101] Preferably, the updating unit is specifically configured to:

[0102] periodically access, by using a crawler technology, a title html selector and a body html selector of each detail page in a database, and when a detail page is accessed from a list page, access, by using the crawler technology, a title html selector and a body html selector of each detail page corresponding to each list item in the list page, and perform page turning according to a page turning operation corresponding to the list page;

[0103] when the title html selector and / or the body html selector of the accessed detail page is invalid, re-determine, by using a large model, whether a page style of the detail page is the same as a page style of a page that has been accessed, if yes, no selector of the detail page needs to be analyzed, and if no, analyze, by using the large model, the selector of the detail page to obtain a title html selector and a body html selector of the detail page, and update the crawler template by using the title html selector and the body html selector of the detail page;

[0104] store, in a database, a page id, a linked url, a list item name, a body, a page state, and a storage time of a detail page in the database that is to be accessed.

[0105] Preferably, the updating unit is further specifically configured to:

[0106] periodically access, by using a crawler technology, each selector of each target sub-page in a database, when a selector of a target sub-page is invalid, determine, by using a large model, a page type of the target sub-page corresponding to the invalid selector, obtain a selector corresponding to the target sub-page corresponding to the invalid selector, and update the crawler template by using the target sub-page corresponding to the invalid selector; or

[0107] periodically access, by using a crawler technology, the network entry page in a database, access, by using the crawler technology, navigation links and related website links of the network entry page, when a new target sub-page exists, determine, by using a large model, a page type of the new target sub-page, obtain a selector corresponding to the new target sub-page, and update the crawler template by using the selector corresponding to the new target sub-page.

[0108] Preferably, the apparatus further comprises a first storage unit configured to store a crawling content table by using a database.

[0109] The crawled content table comprises: a page id corresponding to a detail page, a url of a link corresponding to the detail page, a list item name of a list page corresponding to the detail page, a body of the detail page, a page state of the detail page, and a storage time record of storage of the detail page to a database.

[0110] Preferably, the device further comprises:

[0111] The judging unit is configured to, when accessing a page, if the page returns 404, stop accessing the page, set the page state of the page as invalid, and store the page state of the page to the database; if the page returns error information, re-access the page at a preset time interval, and determine whether the number of times of re-accessing the page reaches a preset number of times; if the page does not return 404 and error information, set the page state of the page as normal, and store the page state of the page to the database.

[0112] If the number of times of re-accessing the page reaches the preset number of times, stop accessing the page, set the page state of the page as invalid, and store the page state of the page to the database; if the number of times of re-accessing the page does not reach the preset number of times, continue to re-access the page at the preset time interval.

[0113] The accessed page comprises the network entry page or the target sub-page.

[0114] Preferably, the device further comprises:

[0115] The second storage unit is configured to store, by using the database, a page style table, a link relationship table of each page, and a storage location of a script file of each page.

[0116] The each page comprises the network entry page or the target sub-page.

[0117] Preferably, the page style table comprises: each type of page style, an id of each type of page style, a storage path of each page corresponding to each type of page style in the database, a url at which each type of page style is stored for the first time, a storage time at which each type of page style is stored to the database, a title html selector of a detail page, a body html selector of the detail page, a list item name selector of a list page, a list item link selector of the list page, a page turning operation of the list page, and whether each page corresponding to the page style is a detail page.

[0118] The link relationship table of each page comprises: a page id of each page, a url of a navigation link and a related website link of each page, a parent link id of the navigation link and the related website link of each page, a page style id corresponding to each page, whether a detail page is unique, a storage time at which each page is stored to the database, and a page state of each page.

[0119] Wherein, the detail page is unique if the detail page is not entered from the list page, otherwise, the detail page is not unique.

[0120] According to a third aspect of the embodiments of the present application, an electronic device is provided, comprising: at least one processor and a memory; the memory and the processor are connected through a bus;

[0121] The memory is configured to store one or more programs.

[0122] When the one or more programs are executed by the at least one processor, the method for determining a large model-based web crawler template is implemented.

[0123] According to a fourth aspect of the embodiments of the present application, a readable storage medium is provided, which has an execution program stored thereon, and the execution program, when executed, implements the method for determining a large model-based web crawler template.

[0124] The technical solutions provided by the present application have the following beneficial effects:

[0125] The method and device for determining a large model-based web crawler template provided by the present application reduce the network page maintenance cost, improve the consistency and standardization level of data processing, improve the automation level and adaptability of data collection, significantly optimize the resource use efficiency, and open up a new path for data acquisition and analysis in the big data era. BRIEF DESCRIPTION OF DRAWINGS

[0126] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiment or prior art description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and those skilled in the art can obtain other drawings according to these drawings without creative labor.

[0127] Figure 1 is a flowchart of a large model-based web crawler template determination method provided by an embodiment of the present application;

[0128] Figure 2 is a schematic diagram of a related website link provided by an embodiment of the present application;

[0129] Figure 3 is a schematic diagram of a list page provided by an embodiment of the present application;

[0130] Figure 4 is a schematic diagram of a list page provided by an embodiment of the present application;

[0131] Figure 5 is a schematic diagram of a list page provided by an embodiment of the present application;

[0132] Figure 6 is a schematic diagram of a list page provided by an embodiment of the present application;

[0133] Figure 7 is a schematic diagram of a list page provided by an embodiment of the present application;

[0134] Figure 8 is a schematic diagram of a list page provided by an embodiment of the present application;

[0135] Figure 9 is a schematic diagram of a list page provided by an embodiment of the present application;

[0136] Figure 10 is a schematic diagram of a list page provided by an embodiment of the present application;

[0137] Figure 11 is a schematic diagram of a list page provided by an embodiment of the present application;

[0138] Figure 12 is a schematic diagram of a list page provided by an embodiment of the present application;

[0139] Figure 13 is a schematic diagram of a list page provided by an embodiment of the present application;

[0140] Figure 14 is a schematic diagram of a list page provided by an embodiment of the present application;

[0141] Figure 15 is a schematic diagram of a list page provided by an embodiment of the present application;

[0142] Figure 16 is a schematic diagram of a list page provided by an embodiment of the present application;

[0143] Figure 17is a schematic view of an example of the selector for analyzing a title and a body provided by an embodiment of the present application;

[0144] Figure 18 is a structural block diagram of a network crawler template determination device based on a large model provided by an embodiment of the present application;

[0145] Figure 19 is a structural block diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0146] In order to make the objects, technical solutions and advantages of the present application clearer, the technical solutions of the present application will be described clearly and completely below with reference to the drawings. Obviously, the following embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0147] Embodiment one

[0148] The present application provides a network crawler template determination method based on a large model, as shown in Figure 1 The method comprises the following steps:

[0149] Step 101: Extract a network entry page from a list of links to be accessed, access the network entry page and extract link-related information of the network entry page, the link-related information comprising navigation links and related website links;

[0150] Step 102: Add the navigation links and the related website links to the list of links to be accessed;

[0151] Step 103: Traverse the navigation links and the related website links in the list of links to be accessed to obtain target subpages;

[0152] Step 104: Analyze the target subpages by using a large model to obtain a crawler template.

[0153] It can be understood that the large model is a large language model (LLM) in full name, which is a super large deep learning model pre-trained based on a large amount of data. Specific examples include ChatGPT and ChatGLM. When a large language model is needed to perform a specific task, it can be trained and fine-tuned with professional data in the relevant field. This stage is based on the pre-training of the model on a large-scale general corpus, aiming to make the model more proficient and adapt to the requirements of this specific task. Through fine-tuning, the model can learn the knowledge, professional terms and subtle differences related to the task in a specific field, thereby significantly improving its analysis and processing capabilities for that task. Once fine-tuning is complete, the trained large language model can efficiently and accurately perform the required task, such as crawler template analysis, page type analysis, etc., providing customized language understanding and generation services for users.

[0154] It should be noted that the "to-be-accessed link list" involved in the embodiments of the present application is preset and therefore is not limited and can be set by those skilled in the art according to engineering needs, experimental data or expert experience, etc.

[0155] Further, the link-related information further includes:

[0156] The navigation bar name, the url corresponding to the navigation link and the url corresponding to the related website link.

[0157] For example, as shown in Figure 2 The related website link can be but is not limited to being set at the bottom of the page.

[0158] Further, step 101 includes:

[0159] Step 1011: extracting the page id and link-related information of the network entry page by using a large model;

[0160] Step 1012: storing the page id and link-related information of the network entry page to a database.

[0161] Further, step 1011 further includes:

[0162] After accessing the network entry page, the script file of the network entry page is obtained;

[0163] The script file of the network entry page is stored to the database.

[0164] It can be understood that the script file is self-contained by the page and can be directly obtained.

[0165] Further, step 101 further includes:

[0166] Step 1013: After extracting the link-related information of the network entry page, analyze the network entry page using a large model to obtain the page style of the network entry page and the page style id;

[0167] Step 1014: Store the page style and page style id of the network entry page to the database.

[0168] Further, step 103 includes:

[0169] Step 1031: Traverse the navigation links and related website links in the list of links to be accessed to obtain the initial subpages corresponding to the navigation links and related website links of the network entry page;

[0170] Step 1032: Remove duplicate pages in the initial subpages to obtain target subpages.

[0171] In some embodiments, the traversal method can be, but is not limited to, breadth-first traversal or depth-first traversal, etc.; each link can also be assigned a weight, and the list can be sorted according to the weight, and the link with the highest weight can be accessed each time.

[0172] Further, step 104 includes:

[0173] Step 1041: Access the target subpage, and determine whether the page style of the target subpage is the same as the page style of the already accessed page. If they are the same, there is no need to determine the page type of the target subpage; if they are not the same, determine the page type of the target subpage using a large model to obtain the page type of the target subpage;

[0174] Step 1042: Based on the page type of the target subpage, analyze the selector of the target subpage using a large model;

[0175] Step 1043: Use the selector of the target subpage to construct a crawler template.

[0176] Further, the determination of the page type of the target subpage using a large model in step 1041 includes:

[0177] When the target subpage includes first information, the page type of the target subpage is a detail page;

[0178] When the target subpage includes second information, the page type of the target subpage is a list page.

[0179] Further, the first information includes at least the following six types: title, text content, publication date, source information, access volume, and responsible editor information;

[0180] The second information includes at least the following three types: list item, list item title, and link to other detailed content.

[0181] Further, step 1042 comprises:

[0182] If the page type of the target sub-page is a list page, the large model is used to analyze the selector of the target sub-page to obtain the list item name selector, the list item link selector, the page style, and the page style id, and the list item name selector, the list item link selector, the page style, and the page style id are stored in the database.

[0183] If the page type of the target sub-page is a detail page, the large model is used to analyze the selector of the target sub-page to obtain the body html selector and the title html selector of the target sub-page, and the body html selector and the title html selector are stored in the database.

[0184] For example, as shown in Figures 3-5 , Figures 3-5 are target sub-pages, it is judged that Figure 3 the page style of the target sub-page is different from the page style of the page that has been visited, the large model is used to judge the page type of the target sub-page, Figure 3 the page type of the target sub-page is a list page, the list item name selector, the list item link selector, the page style, and the page style id are obtained, and the list item name selector, the list item link selector, the page style, and the page style id are stored in the database. Figure 3 It is judged that

[0185] the page style of the target sub-page is different from the page style of the page that has been visited (including the target sub-page that has been visited), Figure 4 the large model is used to judge the page type of the target sub-page, Figure 3 the page type of the target sub-page is a list page, the list item name selector, the list item link selector, the page style, and the page style id are obtained, and the list item name selector, the list item link selector, the page style, and the page style id are stored in the database. Figure 4 Figure 4 It is judged that the page style of the target sub-page is the same as the page style of the target sub-page that has been visited,

[0186] the large model is not needed to judge the page type of the target sub-page, Figure 5 the page style id of the target sub-page is the same as the page style id of the target sub-page that has been visited. Figure 3 Figure 5 Figure 3 Further, step 1041 further comprises:

[0187] Further, step 1042 comprises: ​​

[0188] After accessing the target sub-page, a script file of the target sub-page is obtained;

[0189] The script file of the target sub-page is stored in a database;

[0190] The page id and link-related information of the target sub-page are extracted by using a large model, and are stored in the database;

[0191] The navigation link and the related website link in the link-related information of the target sub-page are added to a list of links to be accessed.

[0192] Further, the method further comprises:

[0193] Step 105: periodically updating a crawler template.

[0194] Further, step 105 comprises:

[0195] Step 1051: periodically traversing and accessing the title html selector and the body html selector of all detail pages in the database by using a crawler technology, and when the detail page is entered from a list page, traversing and accessing the title html selector and the body html selector of the detail page corresponding to each list item in the list page, and performing page turning according to the page turning operation corresponding to the list page;

[0196] Step 1052: when the title html selector and / or the body html selector of the accessed detail page are invalid, using a large model to re-determine whether the page style of the detail page is the same as the page style of the page that has been accessed, if the same, there is no need to analyze the selectors of the detail page; if not the same, using the large model to analyze the selectors of the detail page to obtain the body html selector and the title html selector of the detail page, and using the body html selector and the title html selector of the detail page to update the crawler template;

[0197] Step 1053: storing the page id, the url of the link, the list item name of the list page, the body, the page state and the storage time record time of the detail page in the database into the database.

[0198] In some embodiments, performing page turning according to the page turning operation corresponding to the list page comprises:

[0199] If the page turning operation corresponding to the list page is a js script, a dynamic renderer is started to perform page turning; otherwise, page turning is performed according to the page turning operation corresponding to the list page.

[0200] Further, step 105 further comprises:

[0201] Step 1054: periodically traverse the selectors of each target sub-page in the database using the crawler technology, when there is an invalid selector of the target sub-page, use the large model to judge the page type of the target sub-page corresponding to the invalid selector, obtain the selector corresponding to the target sub-page corresponding to the invalid selector, and update the crawler template using the target sub-page corresponding to the invalid selector; or

[0202] Periodically access the network entry page in the database using the crawler technology, traverse the navigation links and related website links of the network entry page, when there is a new target sub-page, use the large model to judge the page type of the new target sub-page, obtain the selector corresponding to the new target sub-page, and update the crawler template using the selector corresponding to the new target sub-page.

[0203] For example, if the text html selector in the page of the detail page with page style id=3 is changed (for example, from p.content to p.text), the selector soup.select('p.content') will return an empty value when executed, at this time the html needs to be handed over to the large model, let the large model reanalyze the selector, return p.text, and write p.text back to the page style table.

[0204] Further, the method further comprises: step 106: storing the crawling content table in the database;

[0205] The crawling content table includes: the page id corresponding to the detail page, the url of the link corresponding to the detail page, the list item name of the list page corresponding to the detail page, the text of the detail page, the page state of the detail page, and the storage time record time of the detail page stored in the database.

[0206] For example, the selector in the link relationship table is called to crawl the content, for example, the text content of the title 1 (crawling content table page id 1) corresponding to the page id 1 in the link relationship table is crawled, as shown in Figure 6

[0207] from bs4 import BeautifulSoup

[0208] soup = BeautifulSoup(deatil_html, 'html.parser')

[0209] titles = soup.select('p.title') # page type id=3 from page style table

[0210] for title in titles:

[0211] ​write_in_db('title', title.get_text())

[0212] contents = soup.select('p.content') # source from page style table page type id = 3 for content in contents:

[0213] write_in_db('content', content.get_text())

[0214] As Figure 7 shown, the crawled text content is: Today, a test was conducted, and the test was a complete success.

[0215] Further, the method further comprises:

[0216] When accessing the page, if the page returns 404, stop accessing the page, the page state of the page is invalid, and the page state of the page is stored to the database; if the page returns error information, re-access the page every preset time period, and determine whether the number of times of re-accessing the page reaches a preset number of times; if the page does not return 404 and error information, the page state of the page is normal, and the page state of the page is stored to the database;

[0217] If the number of times of re-accessing the page reaches the preset number of times, stop accessing the page, the page state of the page is invalid, and the page state of the page is stored to the database; if the number of times of re-accessing the page does not reach the preset number of times, continue to re-access the page every preset time period;

[0218] The accessed page includes: a network entry page or a target subpage.

[0219] In some embodiments, when the number of times of re-accessing the page reaches the preset number of times, the page is re-accessed after the preset time period is increased, when the number of times of re-accessing the page again reaches the preset number of times, the page is stopped to be accessed, the page state of the page is invalid, and the page state of the page is stored to the database.

[0220] Further, step 106 further comprises:

[0221] The database is used to store the page style table, the link relationship table of each page, and the storage location of the script file of each page;

[0222] The accessed page includes: a network entry page or a target subpage.

[0223] Further, the page style table comprises: each type of page style, an id of each type of page style, a storage path of each page corresponding to each type of page style in the database, a url at which each type of page style is first stored, a storage time at which each type of page style is stored into the database, a title html selector of a detail page, a body html selector of the detail page, a list item name selector of a list page, a list item link selector of the list page, a page turning operation of the list page, and whether each page corresponding to the page style is a detail page;

[0224] The link relationship table of each page comprises: a page id of each page, a url of a navigation link and a related website link of each page, a parent link id of the navigation link and the related website link of each page, a page style id corresponding to each page, whether a detail page is unique, a storage time at which each page is stored into the database, and a page state of each page;

[0225] The detail page is unique if the detail page is not entered from a list page, otherwise, the detail page is not unique.

[0226] It can be understood that the parent link id of the navigation link and the related website link of each page is well known to those skilled in the art, and therefore, the specific implementation mode is not described in detail. For example, if a detail page is entered from a list page, the id of the link of the list page is the parent link id of the detail page.

[0227] It can be understood that the parent link id of the navigation link and the related website link of each page is well known to those skilled in the art, and therefore, the specific implementation mode is not described in detail. For example, if a detail page is entered from a list page, the id of the link of the list page is the parent link id of the detail page.

[0228] For example, the id of the first type of page style is 1, the storage path corresponding thereto in the database is list.html, the url at which it is first stored is http: / / www.test1.cn / list.html, the storage time at which it is stored into the database is January 1, 2024, the list item name selector of the list page is.menu a, and the list item link selector of the list page is.menu b.

[0229] The id of the second type of page style is 2, the storage path corresponding thereto in the database is index.html, the url at which it is first stored is http: / / www.test2.cn / index.html, the storage time at which it is stored into the database is January 2, 2024, the list item name selector of the list page is.bar a, and the list item link selector of the list page is.bar b.

[0230] The third type of page style has an ID of 3, and its corresponding storage path in the database is detail.html. The URL where it was first stored is http: / / www.test1.cn / detail.html, and the storage time in the database is January 3, 2024. The HTML selector for the title of the detail page is p.title, and the HTML selector for the body text of the detail page is P.content.

[0231] To further illustrate the above-mentioned method for determining web crawler templates based on large models, this invention provides a specific example, including the following steps:

[0232] Step 1: Enter the web entry page. This page needs to have a navigation bar link, and may also have links to other websites so that when obtaining the crawler template, all the necessary pages can be accessed.

[0233] Step 2: Access the web entry page, obtain its script file, and store it in the database. Use a large model to extract link-related information from the web entry page, and add the web entry page and its corresponding link information to the list of links to be accessed. Link-related information includes: navigation bar name, navigation links and their corresponding URLs, and related website links and their corresponding URLs. A specific example is shown below:

[0234] Question: Hello, this is the HTML of the webpage. Please extract the navigation bar name and links, as well as related website links.

[0235] Represented in JSON format, such as Figures 8-10 As shown.

[0236] Answer: {"navigationBar":[{"name":"Home","link":"https: / / www.test.cn / index.html"},{"name":"Category 1","link":"https: / / www.test.cn / col1 / index.html"},{"name":"Category 2","link":

[0237] "https: / / www.test.cn / col2 / index.html"},{"name":"Column 3","link":

[0238] "https: / / www.test.cn / col3 / index.html"}],"relatedSites":[{"category":"Unit Type 1","sites":

[0239] [{"name":"Test Unit 1","link":"http: / / www.test1.cn / "},{"name":"Test Unit 2","link":

[0240] The URL is: `"http: / / www.test2.cn / "}]},` where `navigationBar` represents the navigation bar, `relatedSites` represents related links, `name` represents the name, `link` represents the link address, `category` represents the category, and `sites` represents the links for each category.

[0241] Step 3: Analyze the web entry page using a large model to obtain the page style and page style ID of the web entry page, and store the page style and page style ID of the web entry page in the database;

[0242] Step 4: Traverse the navigation links and related website links of the web entry page to obtain the initial subpages corresponding to the navigation links and related website links of the web entry page; remove duplicate pages from the initial subpages to obtain the target subpage;

[0243] Step 5: Access the target subpage, obtain the script file of the target subpage, and store the script file of the target subpage in the database; use the large model to extract the link-related information of the target subpage, and add the corresponding link-related information of the target subpage to the list of links to be accessed;

[0244] Step 6: Determine if the page style of the target subpage is the same as that of the pages that have been visited. If they are the same, there is no need to use the large model to determine the page type of the target subpage; if they are different, use the large model to determine the page type of the target subpage.

[0245] If the page type of the target sub-page is a list page, the selector of the target sub-page is analyzed by using the large model to obtain a list item name selector, a list item link selector, a page turning operation, a page style, and a page style id, and the list item name selector, the list item link selector, the page turning operation, the page style, and the page style id are stored in the database; if the page type of the target sub-page is a detail page, the selector of the target sub-page is analyzed by using the large model to obtain a text html selector and a title html selector of the target sub-page, and the text html selector and the title html selector are stored in the database;

[0246] The list item name selector of the target sub-page, the list item link selector of the target sub-page, the text html selector of the target sub-page, and the title html selector of the target sub-page are used to construct a crawler template;

[0247] Specifically, the page type of the target sub-page is determined by using the large model, including: when the target sub-page includes at least the following six types: title, text content, publication date, source information, access volume, and responsibility editor information, the page type of the target sub-page is a detail page; when the target sub-page includes at least the following three types: list item, list item title, and link to other detailed content, the page type of the target sub-page is a list page;

[0248] The objects for determining the page type mainly include the current page and the reference page, and the reference page is derived from the css and js scripts referenced by the current page, as shown in the following figure. Figure 11

[0249] For the referenced scripts, invisible characters such as spaces and line breaks need to be removed, and utf-8 encoding is used to calculate the edit distance with the existing non-failed scripts in the database, and the script with the minimum value is obtained, and the edit distance calculation is as follows, where a and b represent the contents of the current script and the script in the database respectively.

[0250] If |a| and |b| represent the lengths of the two strings a and b respectively, then their Levenshtein distance is lev a,b (|a|,|b|), which satisfies:

[0251]

[0252] In the above formula, i∈[1,|a|], j∈[1,|b|], is an indicator function, and the value of i ≠b j , is 0, otherwise, the value of a,b ​(i,j) is the Levenshtein distance between the first i characters of string a and the first j characters of string b (i and j are lower indices starting at 1); lev a,b (i-1,j) is the Levenshtein distance between the first i-1 characters of string a and the first j characters of string b, lev a,b (i,j-1) is the Levenshtein distance between the first i characters of string a and the first j-1 characters of string b, lev a,b (i-1,j-1) is the Levenshtein distance between the first i-1 characters of string a and the first j-1 characters of string b, Note that in the min operation, the first formula represents deleting characters (from a) to reach b, the second formula represents inserting characters, and the third formula represents replacement (depending on whether the current characters are the same);

[0253] For the current page (html), in addition to removing invisible characters such as spaces and line breaks, html comments, link content (href attribute in a tag, src attribute in img tag), meta tags, and text in tags, use utf-8 encoding, and calculate the edit distance with the existing non-expired pages in the database to get the minimum value page; The total distance calculation formula is as follows:

[0254]

[0255] where e ∈ [1, n], n is the total number of reference scripts, is the average distance of the reference script, d e is the distance of the e-th reference script, w a is the distance weight of the reference script, w b is the distance weight of the current page, d b is the distance of the current page; where w a +w b = 1, when the total distance is less than the set threshold, it is judged that the page style is similar;

[0256] Examples of large model analysis of whether the page is a list page or a detail page are as follows:

[0257] Question: Hello, this is a web page html, please analyze whether the page is a list page or a detail page. Use json to represent, as shown in Figure 12 and Figure 13 .

[0258] Answer: According to the analysis of the provided HTML code, this page is a detail page. The following points support this judgment:

[0259] The page contains a clear title ( <h2 class＝"article-subtitle title">), usually with a title of the specific content;

[0260] There is a date of publication ( <span class＝"date">Time:

[0261] <publishtime> 2000-01-01< / publishtime> ) and source information (<spanclass="ly">Source: Unit 1), which is an element that is often found on the details page, used to mark the specific information of the article or content;

[0262] The main part of the page contains detailed text content ( <div class＝"article-content"id＝"zoomcon">), which displays specific paragraph information, and this is the manifestation of the details content;

[0263] There is a display of access volume ( <span class＝"llcs">Access: <b id＝"view">1 ), which is usually used to count the browsing of individual content pages;

[0264] There is "editor in charge" information at the bottom of the page ( <span class＝"editer">Editor in charge: Personnel ), which is the copyright or editor information part commonly seen in content detail pages;

[0265] There is a lack of common features of list pages, such as multiple list items, list item titles, and links to summaries of other detailed content, etc.

[0266] If it is a list page, record the list name and link selector as well as the page turning method, and enter a detail page, as shown in the following example:

[0267] Question: Hello, the following is the html of the webpage, please analyze the list title and link selector, and analyze the page turning method. Use json to represent, as shown in Figure 14 and Figure 15 .

[0268] Answer: ### List title and link selector analysis:

[0269] 1. **List title selector**:

[0270] - The title in the list item is wrapped in ` Text within tags, each heading is located in a div with class `.list` under the label Within the `<list>` element. Therefore, to select all list headings, you can use the following CSS selector:

[0271] ```

[0272] CSS

[0273] .list li a

[0274] ```

[0275] This selector will match all links and their text content in all list items;

[0276] 2. **Link Selector**:

[0277] - Specifically for retrieving the link to each item in the list (assuming links need to be handled individually), the same selector `.list li a` works, but if you want to manipulate the link address, you can further extract or manipulate the `href` attribute using JavaScript or other programming languages.

[0278] ###Analysis of Page Turning Methods:

[0279] The page-turning function is implemented through pagination navigation, which mainly includes the following parts:

[0280] 1. **Previous Page**:

[0281] - Link selector: `.newspage ul li:first-child a`

[0282] - This selector will locate the link to the "Previous Page" button;

[0283] 2. **Page Number**:

[0284] - The current page number is not clickable, but you can jump to the next or previous page through the page number links around it.

[0285] - Regular page number link selector: `.newspage ul li:not(:first-child):not(:last-child)a`

[0286] - Select all page number links except for "Previous Page" and "Next Page";

[0287] 3. **Next Page**:

[0288] - Link selector: `.newspage ul li.next a`

[0289] - A link to the "Next Page" button;

[0290] 4. Jump to specified page:

[0291] - Click the "OK" button after entering the page number to jump;

[0292] - Input box selector: `#pnum`

[0293] - Confirm button selector: `.newspageBtn`

[0294] Through analysis, users can click "Previous Page", "Next Page" buttons or directly click the page number to switch pages, or enter the specified page number and click "OK" button to jump to the corresponding page. In actual operation, JavaScript can be used to listen to the click events of these elements, read the value of the input box, and then update the URL or perform AJAX request to achieve dynamic loading or jumping of page content;

[0295] If it is a detail page, the large model analyzes the html selectors of the title and body and records them in the database. For example, the question is: Hello, this is a web page HTML, please analyze the selectors of the title and body. Use JSON to represent, as shown in Figure 16 and Figure 17 .

[0296] The answer is as follows: According to the HTML code you provided, the selectors for the title and body are as follows:

[0297] Title selector: To select the title of the page, you can use the following CSS selector: h2.article-subtitle.title, which directly locates to <h2>Tags, with elements of class.article-subtitle and.title, i.e. the main title of the page;

[0298] Body selector: The body content is contained in a series of within tags, and these tags are located in a div with class.article-content Thus, the selector for the body content is: #zoomcon p, which selects the element with the ID of zoomcon all of the interior The tags cover all body paragraphs from "Paragraph 1" to "Paragraph 3";

[0299] These two selectors can be used to locate and manipulate the title and body content using JavaScript or CSS, respectively.

[0300] Step 7: When accessing the page, if the page returns a 404 error, stop accessing the page, set the page status to invalid, and store the page status in the database; if the page returns an error message, re-access the page at preset time intervals and determine whether the number of re-accesses has reached the preset number.

[0301] If the number of times the page is accessed reaches the preset number, access to the page will stop, the page status will be invalid, and the page status will be stored in the database; if the number of times the page is accessed does not reach the preset number, the page will continue to be accessed until the number of times the page is accessed reaches the preset number or the page does not return an error message, the page status will be normal, and the page status will be stored in the database.

[0302] The pages accessed include: the web entry page or the target subpage;

[0303] Step 8: Use a database to store the page style table, the link relationship table of each page, and the storage location of the script files for each page; where each page includes: the web entry page or the target subpage; the script files for each page include: html and / or css and / or js script files;

[0304] Step 9: Regularly update the crawler template, which can be done through partial updates or global updates;

[0305] Partial updates include: periodically traversing and accessing the title HTML selector and body HTML selector of all detail pages in the database using web crawling technology; and when a detail page is accessed from a list page, traversing and accessing the title HTML selector and body HTML selector of the detail page corresponding to each list item in the list page, and performing page turning according to the page turning operation corresponding to the list page.

[0306] When the title HTML selector and / or body HTML selector of the visited detail page are invalid, the large model is used to re-determine whether the page style of the detail page is the same as the page style of the visited page. If they are the same, there is no need to analyze the selector of the detail page; if they are not the same, the large model is used to analyze the selector of the detail page to obtain the body HTML selector and title HTML selector of the detail page, and the crawler template is updated using the body HTML selector and title HTML selector of the detail page.

[0307] Wherein, the global update period can be but not limited to one month to two months, and the update time and page state in the update database are updated after the update.

[0308] The global update includes two ways: first, using the crawler technology to periodically access each selector of each target sub-page in the database, when there is an invalid selector of the target sub-page, using the large model to judge the page type of the target sub-page corresponding to the invalid selector, obtaining the selector corresponding to the target sub-page corresponding to the invalid selector, and updating the crawler template using the target sub-page corresponding to the invalid selector; second, using the crawler technology to periodically access the network entry page in the database, traversing the navigation link and related website link of the network entry page, when there is a new target sub-page, using the large model to judge the page type of the new target sub-page, obtaining the selector corresponding to the new target sub-page, and updating the crawler template using the selector corresponding to the new target sub-page.

[0309] In the face of the challenges of frequent webpage versioning and variable structure, the large model-based network crawler updating method provided by the application significantly improves the efficiency and flexibility of data collection, bringing a series of positive effects.

[0310] Firstly, the application greatly reduces the maintenance workload of the crawler crawling the crawler template. By integrating the large model, the crawler can automatically understand and adapt to the changes in the webpage structure, intelligently identify and extract key information, without the need for frequent manual adjustment of parsing rules. This not only saves valuable development resources, but also speeds up the response speed to newly revised websites, ensuring the continuity of data collection tasks.

[0311] Secondly, it effectively reduces the workload of manual paging and monitoring. Traditional crawlers often need to design special paging logic or monitoring mechanisms for different pages, while under the support of the large model, the crawler can automatically identify and execute paging operations, even predict and follow links to deep pages, greatly expanding the breadth and depth of data collection, while liberating human resources.

[0312] Thirdly, the application can effectively classify and integrate different page styles and layouts. Each access saves the webpage html, and each access to a new page compares the similarity with existing pages to filter out pages of the same style, reducing the time spent accessing the large model, saving the cost of large model calls, and improving the consistency and standardization level of data processing.

[0313] In summary, the large model-based network crawler template determination method provided by the application effectively reduces the artificial maintenance cost brought by webpage versioning in the context of massive webpage data, not only improves the automation level and adaptability of data collection, but also significantly optimizes the resource utilization efficiency, opening up a new path for data acquisition and analysis in the big data era.

[0314] Embodiment Two

[0315] The application also provides a large model-based web crawler template determination device, as shown in the accompanying drawings, comprising: Figure 18

[0316] An extraction unit is configured to extract a web portal page from a to-be-visited link list, visit the web portal page, and extract link-related information of the web portal page, the link-related information comprising navigation links and related website links;

[0317] An adding unit is configured to add the navigation links and the related website links to the to-be-visited link list;

[0318] A first obtaining unit is configured to traverse the navigation links and the related website links in the to-be-visited link list to obtain a target subpage;

[0319] A second obtaining unit is configured to analyze the target subpage by using a large model to obtain a crawler template.

[0320] Further, the link-related information further comprises:

[0321] a navigation bar name, a url corresponding to the navigation links, and a url corresponding to the related website links.

[0322] Further, the extraction unit comprises:

[0323] A first extraction module is configured to extract a page id of the web portal page and the link-related information by using a large model;

[0324] A first storage module is configured to store the page id of the web portal page and the link-related information into a database.

[0325] Further, the first extraction module is further configured to:

[0326] obtain a script file of the web portal page after visiting the web portal page;

[0327] store the script file of the web portal page into the database.

[0328] Further, the extraction unit further comprises:

[0329] A first analysis module is configured to analyze the web portal page by using a large model to obtain a page style and a page style id of the web portal page after extracting the link-related information of the web portal page;

[0330] A second storage module is configured to store the page style and the page style id of the web portal page into the database.

[0331] ​Further, the first obtaining unit comprises:

[0332] The first obtaining module is configured to traverse the navigation links and the related website links in the list of to-be-accessed links to obtain the navigation links and the related website links of the network entry page corresponding to initial subpages;

[0333] The second obtaining module is configured to eliminate duplicate pages in the initial subpages to obtain target subpages.

[0334] Further, the second obtaining unit comprises:

[0335] The determining module is configured to access the target subpages, determine whether the page styles of the target subpages are the same as the page styles of the already accessed pages, if the page styles are the same, it is not necessary to determine the page types of the target subpages, and if the page styles are not the same, it is determined, by using the large model, the page types of the target subpages to obtain the page types of the target subpages.

[0336] The second analysis module is configured to analyze, based on the page types of the target subpages, the selectors of the target subpages by using the large model.

[0337] The constructing module is configured to construct a crawler template by using the selectors of the target subpages.

[0338] Further, the determining module is further configured to:

[0339] When the target subpages include the first information, the page types of the target subpages are detail pages.

[0340] When the target subpages include the second information, the page types of the target subpages are list pages.

[0341] Further, the first information at least includes the following six kinds: a title, text content, a publication date, source information, a number of visits and responsibility editor information.

[0342] The second information at least includes the following three kinds: a list item, a list item title and a link to other detailed content.

[0343] Further, the second analysis module is specifically configured to:

[0344] If the page types of the target subpages are list pages, the selectors of the target subpages are analyzed by using the large model to obtain a list item name selector, a list item link selector, a page turning operation, a page style and a page style id, and the list item name selector, the list item link selector, the page turning operation, the page style and the page style id are stored in a database;

[0345] If the page type of the target sub-page is a detail page, a large model is used to analyze the selector of the target sub-page to obtain a body html selector and a title html selector of the target sub-page, and the body html selector and the title html selector are stored in the database.

[0346] Further, the judging module is further configured to:

[0347] After accessing the target sub-page, a script file of the target sub-page is obtained.

[0348] The script file of the target sub-page is stored in the database.

[0349] A page id and link related information of the target sub-page are extracted by using a large model, and are stored in the database.

[0350] Navigation links and related website links in the link related information of the target sub-page are added to a to-be-accessed link list.

[0351] Further, the device further comprises:

[0352] An updating unit is configured to update the crawler template periodically.

[0353] Further, the updating unit is specifically configured to:

[0354] The title html selector and the body html selector of all detail pages in the database are accessed periodically by using a crawler technology, and when the detail pages are entered from a list page, the title html selector and the body html selector of the detail pages corresponding to each list item in the list page are accessed, and the pages are turned according to a page turning operation of the list page;

[0355] When the title html selector and / or the body html selector of the accessed detail page are invalid, a large model is used to rejudge whether the page style of the detail page is the same as the page style of a page that has been accessed, if the page styles are the same, the selector of the detail page does not need to be analyzed, if the page styles are different, the selector of the detail page is analyzed by using a large model to obtain a body html selector and a title html selector of the detail page, and the body html selector and the title html selector of the detail page are used to update the crawler template.

[0356] The page id, the url of the link, the list item name of the list page, the body, the page state, and the storage time record time of the detail page in the database that are stored in the database are stored in the database.

[0357] Further, the updating unit is further specifically configured to:

[0358] The selector of each target sub-page in the database is periodically accessed by using a crawler technology. When the selector of the target sub-page is invalid, a large model is used to determine the page type of the target sub-page corresponding to the invalid selector, obtain the selector corresponding to the target sub-page corresponding to the invalid selector, and update the crawler template by using the target sub-page corresponding to the invalid selector.

[0359] The network entry page in the database is periodically accessed by using a crawler technology. The navigation link and the related website link of the network entry page are traversed. When a new target sub-page exists, a large model is used to determine the page type of the new target sub-page, obtain the selector corresponding to the new target sub-page, and update the crawler template by using the selector corresponding to the new target sub-page.

[0360] Further, the device further comprises a first storage unit configured to store the crawling content table in the database.

[0361] The crawling content table comprises a page id corresponding to a detail page, a url of a link corresponding to the detail page, a list item name of a list page corresponding to the detail page, a body of the detail page, a page state of the detail page, and a storage time record time of the detail page stored in the database.

[0362] Further, the device further comprises:

[0363] The judging unit is configured to, when accessing a page, if the page returns 404, stop accessing the page, the page state of the page is invalid, and the page state of the page is stored in the database; if the page returns error information, re-access the page every preset time period, and determine whether the number of times of re-accessing the page reaches a preset number of times; if the page does not return 404 and error information, the page state of the page is normal, and the page state of the page is stored in the database.

[0364] If the number of times of re-accessing the page reaches the preset number of times, stop accessing the page, the page state of the page is invalid, and the page state of the page is stored in the database; if the number of times of re-accessing the page does not reach the preset number of times, continue to re-access the page every preset time period.

[0365] The accessed page comprises a network entry page or a target sub-page.

[0366] Further, the device further comprises:

[0367] The second storage unit is configured to store a page style table, a link relationship table of each page, and a script file storage location of each page in the database.

[0368] The accessed page comprises a network entry page or a target sub-page.

[0369] Further, the page style table comprises: each type of page style, an id of each type of page style, a storage path of each page corresponding to each type of page style in the database, a url at which each type of page style is stored for the first time, a storage time at which each type of page style is stored into the database, a title html selector of a detail page, a body html selector of the detail page, a list item name selector of a list page, a list item link selector of the list page, a page turning operation of the list page, and whether each page corresponding to the page style is a detail page;

[0370] The link relationship table of each page comprises: a page id of each page, a url of a navigation link and a related website link of each page, a parent link id of the navigation link and the related website link of each page, a page style id corresponding to each page, whether a detail page is unique, a storage time at which each page is stored into the database, and a page state of each page;

[0371] The detail page is unique if the detail page is not entered from a list page, and the detail page is not unique otherwise.

[0372] It can be understood that the device embodiments provided above correspond to the method embodiments described above, and the corresponding specific contents can be mutually referred to, and will not be described here in detail.

[0373] It can be understood that the same or similar parts in the above embodiments can be mutually referred to, and the contents not described in detail in some embodiments can be referred to the same or similar contents in other embodiments.

[0374] Embodiment Three

[0375] As shown in Figure 19 The electronic device in the embodiment can comprise a processor, a memory, a transceiver component, etc. The memory, the processor and the transceiver component are connected through a bus; the memory can be used to store an execution program, and the exemplary execution program can comprise instructions; the processor is used to execute the instructions stored in the memory. The memory can also be used to store data, which can be called and / or modified when the instructions are executed.

[0376] The processor can be a central processing unit (CPU), and can also be other general-purpose processors, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, etc., which are the computing core and control core of the terminal, and are suitable for implementing one or more instructions, and are specifically suitable for loading and executing one or more instructions in the storage medium to implement a corresponding method flow or a corresponding function, to implement the steps of the network crawler template determination method based on a large model in the above embodiment.

[0377] Embodiment four

[0378] Based on the same inventive concept, the application further provides a readable storage medium, specifically an electronic device readable storage medium (Memory). The electronic device readable storage medium is a memory device in the electronic device, and is used to store programs and data. It can be understood that the storage medium herein can include a built-in storage medium in the electronic device, and of course can also include an expansion storage medium supported by the electronic device. The storage medium provides a storage space, and the storage space stores an operating system of the terminal. In addition, one or more instructions suitable for being loaded and executed by the processor are also stored in the storage space, and these instructions can be one or more execution programs (including program codes). It should be noted that the storage medium herein can be a high-speed RAM memory, or a non-volatile memory such as at least one disk memory. Loading and executing one or more instructions stored in the storage medium by the processor can implement the steps of the network crawler template determination method based on a large model in the above embodiment.

[0379] Those skilled in the art should understand that the embodiments of the application can be provided as a method, a system, or a computer program product. Therefore, the application can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the application can take the form of a computer program product implemented on one or more computer usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer usable program code.

[0380] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart or flows and / or block diagram block or blocks. Figure One one or more flow or flows and / or block diagram block or blocks. Figure One one or more flow or flows and / or block diagram block or blocks.

[0381] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart or flows and / or block diagram block or blocks. Figure One one or more flow or flows and / or block diagram block or blocks. Figure One one or more flow or flows and / or block diagram block or blocks.

[0382] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart or flows and / or block diagram block or blocks. Figure One one or more flow or flows and / or block diagram block or blocks. Figure One one or more flow or flows and / or block diagram block or blocks.

[0383] Finally, it should be noted that the above-mentioned embodiments are merely intended to illustrate the technical solutions of the present application, rather than limit the same. Although the present application has been described in detail with reference to the above-mentioned embodiments, those skilled in the art should understand that the specific embodiments of the present application can be modified or equivalent replacements without departing from the spirit and scope of the present application, and any modifications or equivalent replacements shall be included in the protection scope of the claims of the present application. < / h2> < / h2>

Claims

1. A method for determining web crawler templates based on large models, characterized in that, include: Extract the network entry page from the list of links to be accessed, access the network entry page and extract the link-related information of the network entry page, the link-related information including: navigation links and related website links; Add the navigation links and the relevant website links to the list of links to be visited; Iterate through the navigation links and related website links in the list of links to be accessed to obtain the target subpage; The target subpage is analyzed using a large model to obtain the crawler template.

2. The method according to claim 1, characterized in that, The link-related information also includes: Navigation bar name, URLs corresponding to navigation links, and URLs corresponding to related website links.

3. The method according to claim 1, characterized in that, The extraction of link-related information from the network entry page includes: The page ID and link information of the network entry page are extracted using a large model; Store the page ID and link information of the network entry page in the database.

4. The method according to claim 1, characterized in that, Also includes: After accessing the network entry page, obtain the script file of the network entry page; Store the script file of the web entry page in the database.

5. The method according to claim 1, characterized in that, Also includes: After extracting the link-related information from the web entry page, the web entry page is analyzed using a large model to obtain the page style and page style ID of the web entry page; Store the page style and page style ID of the web entry page in the database.

6. The method according to claim 1, characterized in that, The process of traversing the navigation links and related website links in the list of links to be accessed to obtain the target subpage includes: Traverse the navigation links and related website links in the list of links to be accessed to obtain the initial subpages corresponding to the navigation links and related website links of the network entry page; The target subpage is obtained by removing duplicate pages from the initial subpage.

7. The method according to claim 1, characterized in that, The process of analyzing the target subpage using a large model to obtain a crawler template includes: Access the target subpage and determine whether the page style of the target subpage is the same as the page style of the pages that have been accessed. If they are the same, there is no need to determine the page type of the target subpage. If they are different, the page type of the target subpage is determined using the large model. Based on the page type of the target subpage, the selector of the target subpage is analyzed using a large model; The crawler template is constructed using the selector of the target subpage.

8. The method according to claim 7, characterized in that, The process of determining the page type of the target subpage using a large model includes: When the target subpage includes the first information, the page type of the target subpage is a details page; When the target subpage includes the second information, the page type of the target subpage is a list page.

9. The method according to claim 8, characterized in that, The first information includes at least the following six types: title, text content, publication date, source information, number of visits, and editor information; The second information includes at least the following three types: list items, list item titles, and summaries linking to other details.

10. The method according to claim 7, characterized in that, The step of analyzing the selector of the target subpage based on its page type using a large model includes: If the target subpage is a list page, then the selector of the target subpage is analyzed using a large model to obtain the list item name selector, list item link selector, pagination operation, page style and page style ID, and the list item name selector, the list item link selector, the pagination operation, the page style and the page style ID are stored in the database. If the target subpage is a details page, then the selector of the target subpage is analyzed using a large model to obtain the body HTML selector and the title HTML selector of the target subpage, and the body HTML selector and the title HTML selector are stored in the database.

11. The method according to claim 7, characterized in that, Also includes: After accessing the target subpage, obtain the script file of the target subpage; Store the script file of the target subpage in the database; The page ID and link-related information of the target subpage are extracted using a large model and stored in the database; Add the navigation links and related website links from the link information of the target subpage to the list of links to be visited.

12. The method according to claim 9, characterized in that, Also includes: The crawler template is updated regularly.

13. The method according to claim 12, characterized in that, The periodic updating of the crawler template includes: Using web crawling technology, the title HTML selector and body HTML selector of all detail pages in the database are periodically traversed and accessed. When the detail page is entered from the list page, the title HTML selector and body HTML selector of the detail page corresponding to each list item in the list page are traversed and accessed, and the page is turned according to the page turning operation corresponding to the list page. When the title HTML selector and / or body HTML selector of the accessed details page are invalid, the large model is used to re-determine whether the page style of the details page is the same as the page style of the previously accessed pages. If they are the same, there is no need to analyze the selector of the details page; if they are not the same, the large model is used to analyze the selector of the details page to obtain the body HTML selector and title HTML selector of the details page, and the crawler template is updated using the body HTML selector and title HTML selector of the details page. The page ID corresponding to the detail page, the URL of the link, the list item name of the list page, the content, the page status, and the storage time record of the database are stored in the database.

14. The method according to claim 12, characterized in that, The periodic updating of the crawler template also includes: Using web crawling technology, the crawler periodically traverses and accesses each selector of each target subpage in the database. When an invalid selector is found for a target subpage, a large model is used to determine the page type of the target subpage corresponding to the invalid selector, the selector corresponding to the target subpage of the invalid selector is obtained, and the crawler template is updated using the target subpage corresponding to the invalid selector; or The web crawler uses web crawling technology to periodically access the network entry pages in the database, traverse the navigation links and related website links of the network entry pages, and when a new target subpage is found, it uses a large model to determine the page type of the new target subpage, obtains the selector corresponding to the new target subpage, and updates the web crawler template using the selector corresponding to the new target subpage.

15. The method according to claim 13, characterized in that, This also includes: using a database to store the crawled content table; The crawled content table includes: the page ID corresponding to the details page, the URL of the link corresponding to the details page, the list item name of the list page corresponding to the details page, the main content of the details page, the page status of the details page, and the storage time record of the details page being stored in the database.

16. The method according to claim 1 or 7, characterized in that, Also includes: When accessing a page, if the page returns a 404 error, access to the page is stopped, the page status is set to invalid, and the page status is stored in the database. If the page returns an error message, the page is revisited every preset time period, and it is determined whether the number of times the page is revisited has reached the preset number. If the page does not return a 404 error message, the page status is normal, and the page status is stored in the database. If the number of times the page is accessed reaches a preset number, access to the page will stop, the page status will be invalid, and the page status will be stored in the database. If the number of times the page is revisited does not reach the preset number, the page will continue to be revisited every preset time period. The accessed pages include: the network entry page or the target subpage.

17. The method according to claim 1, characterized in that, Also includes: The database is used to store the page style table, the link relationship table of each page, and the storage location of the script files of each page; The pages mentioned include: the network entry page or the target subpage.

18. The method according to claim 17, characterized in that, The page style table includes: page styles of each type, the ID of each page style of each type, the storage path of each page corresponding to each page style in the database, the URL where each page style of each type is first stored, the storage time when each page style of each type is stored in the database, the HTML selector for the title of the detail page, the HTML selector for the body of the detail page, the list item name selector for the list page, the list item link selector for the list page, the pagination operation of the list page, and whether each page corresponding to the page style is a detail page; The link relationship table for each page includes: the page ID of each page, the URLs of the navigation links and related website links of each page, the parent link IDs of the navigation links and related website links of each page, the page style ID corresponding to each page, whether the details page is unique, the storage time of each page in the database, and the page status of each page. The uniqueness of a details page is determined by whether the details page is unique if it is not accessed from the list page; otherwise, the details page is not unique.

19. A web crawler template determination device based on a large model, characterized in that, include: The extraction unit is used to extract network entry pages from the list of links to be accessed, access the network entry pages and extract the link-related information of the network entry pages, the link-related information including: navigation links and related website links; An adding unit is used to add the navigation links and the related website links to the list of links to be accessed; The first acquisition unit is used to traverse the navigation links and related website links in the list of links to be accessed to obtain the target subpage; The second acquisition unit is used to analyze the target sub-page using a large model to obtain a crawler template.

20. The apparatus according to claim 19, characterized in that, The link-related information also includes: Navigation bar name, URLs corresponding to navigation links, and URLs corresponding to related website links.

21. The apparatus according to claim 19, characterized in that, The extraction unit includes: The first extraction module is used to extract the page ID and link-related information of the network entry page using a large model; The first storage module is used to store the page ID and link-related information of the network entry page into the database.

22. The apparatus according to claim 21, characterized in that, The first extraction module is further configured to: After accessing the network entry page, obtain the script file of the network entry page; Store the script file of the web entry page in the database.

23. The apparatus according to claim 19, characterized in that, The extraction unit further includes: The first analysis module is used to analyze the web entry page using a large model after extracting the link-related information of the web entry page, and to obtain the page style and page style ID of the web entry page. The second storage module is used to store the page style and page style ID of the network entry page into the database.

24. The apparatus according to claim 19, characterized in that, The first acquisition unit includes: The first acquisition module is used to traverse the navigation links and related website links in the list of links to be accessed, and obtain the initial subpages corresponding to the navigation links and related website links of the network entry page; The second acquisition module is used to remove duplicate pages from the initial subpages to obtain the target subpage.

25. The apparatus according to claim 19, characterized in that, The second acquisition unit includes: The judgment module is used to access the target subpage and determine whether the page style of the target subpage is the same as the page style of the pages that have been accessed. If they are the same, there is no need to determine the page type of the target subpage; if they are not the same, the page type of the target subpage is determined by using the large model. The second analysis module is used to analyze the selector of the target subpage based on the page type of the target subpage using a large model; A building module is used to construct the crawler template using the selector of the target subpage.

26. The apparatus according to claim 25, characterized in that, The judgment module is also used for: When the target subpage includes the first information, the page type of the target subpage is a details page; When the target subpage includes the second information, the page type of the target subpage is a list page.

27. The apparatus according to claim 26, characterized in that, The first information includes at least the following six types: title, text content, publication date, source information, number of visits, and editor information; The second information includes at least the following three types: list items, list item titles, and summaries linking to other details.

28. The apparatus according to claim 25, characterized in that, The second analysis module is specifically used for: If the target subpage is a list page, then the selector of the target subpage is analyzed using a large model to obtain the list item name selector, list item link selector, pagination operation, page style and page style ID, and the list item name selector, the list item link selector, the pagination operation, the page style and the page style ID are stored in the database. If the target subpage is a details page, then the selector of the target subpage is analyzed using a large model to obtain the body HTML selector and the title HTML selector of the target subpage, and the body HTML selector and the title HTML selector are stored in the database.

29. The apparatus according to claim 25, characterized in that, The judgment module is also used for: After accessing the target subpage, obtain the script file of the target subpage; Store the script file of the target subpage in the database; The page ID and link-related information of the target subpage are extracted using a large model and stored in the database; Add the navigation links and related website links from the link information of the target subpage to the list of links to be visited.

30. The apparatus according to claim 27, characterized in that, The device further includes: An update unit is used to periodically update the crawler template.

31. The apparatus according to claim 30, characterized in that, The update unit is specifically used for: Using web crawling technology, the title HTML selector and body HTML selector of all detail pages in the database are periodically traversed and accessed. When the detail page is entered from the list page, the title HTML selector and body HTML selector of the detail page corresponding to each list item in the list page are traversed and accessed, and the page is turned according to the page turning operation corresponding to the list page. When the title HTML selector and / or body HTML selector of the accessed details page are invalid, the large model is used to re-determine whether the page style of the details page is the same as the page style of the previously accessed pages. If they are the same, there is no need to analyze the selector of the details page. If they are different, the selector of the details page is analyzed using a large model to obtain the body HTML selector and title HTML selector of the details page, and the crawler template is updated using the body HTML selector and title HTML selector of the details page. The page ID corresponding to the detail page, the URL of the link, the list item name of the list page, the content, the page status, and the storage time record of the database are stored in the database.

32. The apparatus according to claim 30, characterized in that, The update unit is also specifically used for: The crawler uses web crawling technology to periodically traverse and access each selector of each target subpage in the database. When a selector of a target subpage is invalid, the crawler uses a large model to determine the page type of the target subpage corresponding to the invalid selector, obtains the selector corresponding to the target subpage corresponding to the invalid selector, and updates the crawler template using the target subpage corresponding to the invalid selector. or The web crawler uses web crawling technology to periodically access the network entry pages in the database, traverse the navigation links and related website links of the network entry pages, and when a new target subpage is found, it uses a large model to determine the page type of the new target subpage, obtains the selector corresponding to the new target subpage, and updates the web crawler template using the selector corresponding to the new target subpage.

33. The apparatus according to claim 31, characterized in that, The device further includes: a first storage unit for storing a crawled content table using a database; The crawled content table includes: the page ID corresponding to the details page, the URL of the link corresponding to the details page, the list item name of the list page corresponding to the details page, the main content of the details page, the page status of the details page, and the storage time record of the details page being stored in the database.

34. The apparatus according to claim 19 or 25, characterized in that, The device further includes: The judgment unit is configured to, when accessing a page, if the page returns a 404 error, stop accessing the page, set the page status to invalid, and store the page status in the database; if the page returns an error message, re-access the page at preset time intervals and determine whether the number of re-accesses has reached the preset number; if the page does not return a 404 error or an error message, the page status is normal, and the page status is stored in the database. If the number of times the page is revisited reaches a preset number, access to the page will stop, the page status will be invalid, and the page status will be stored in the database; if the number of times the page is revisited does not reach the preset number, the page will continue to be revisited every preset time period. The accessed pages include: the network entry page or the target subpage.

35. The apparatus according to claim 19, characterized in that, The device further includes: The second storage unit is used to store the page style table, the link relationship table of each page, and the storage location of the script files of each page using a database. The pages mentioned include: the network entry page or the target subpage.

36. The apparatus according to claim 35, characterized in that, The page style table includes: page styles of each type, the ID of each page style of each type, the storage path of each page corresponding to each page style in the database, the URL where each page style of each type is first stored, the storage time when each page style of each type is stored in the database, the HTML selector for the title of the detail page, the HTML selector for the body of the detail page, the list item name selector for the list page, the list item link selector for the list page, the pagination operation of the list page, and whether each page corresponding to the page style is a detail page; The link relationship table for each page includes: the page ID of each page, the URLs of the navigation links and related website links of each page, the parent link IDs of the navigation links and related website links of each page, the page style ID corresponding to each page, whether the details page is unique, the storage time of each page in the database, and the page status of each page. The uniqueness of a details page is determined by whether the details page is unique if it is not accessed from the list page; otherwise, the details page is not unique.

37. An electronic device, characterized in that, include: At least one processor and memory; The memory and processor are connected via a bus; The memory is used to store one or more programs; When the one or more programs are executed by the at least one processor, the method for determining web crawler templates based on large models as described in any one of claims 1 to 18 is implemented.

38. A readable storage medium, characterized in that, It contains an executable program, which, when executed, implements the web crawler template determination method based on a large model as described in any one of claims 1 to 18.

Images