Network data adaptive acquisition method and system based on pre-training large model

Abstract

The invention provides a network data self-adaptive acquisition method and system based on a pre-trained large model, and the method comprises the steps: adding an HTML tag and a self-defined attribute in webpage data into a vocabulary of a Token initiator, and initializing an embedded layer of the pre-trained large model; performing incremental training and supervised fine-tuning training on the pre-trained large model by utilizing the preprocessed model training sample and the vocabulary to generate a network structure recognition large model, analyzing input webpage data to obtain webpage structured data, and regularly obtaining screenshots of a target webpage according to a preset time interval. Performing difference comparison on the webpage structured data and the webpage screenshots, generating a difference list through a difference detection algorithm, and calculating a comprehensive confidence score; and generating a specific crawler adjustment strategy according to the difference list, and updating the existing crawler rule. The method has stronger generalization and self-adaption capabilities, can automatically adapt to frequently changing webpage structures, and improves the efficiency and accuracy of data acquisition.

Description

Technical Field

[0001] The present invention relates to the field of computer and data processing technology, and in particular to a network data adaptive collection method and system based on a pre-trained large model. Background Art

[0002] In the collection of network data, due to the frequent changes in the structure and layout of web pages, especially the embedding of dynamic pages (such as advertisements, verification codes, etc.), the collection program often fails to collect according to the preset rules and causes interruptions.

[0003] The current web data collection methods mainly include:

[0004] 1. Extract web page content through manually set CSS selectors, XPath and other rules, but this method is very sensitive to changes in web page structure and has high maintenance costs;

[0005] 2. Match and extract web page content through pre-defined templates, but lack flexibility and have difficulty dealing with complex and dynamically changing web pages;

[0006] 3. Use machine learning models to identify web page elements, but they are usually limited to specific fields or specific types of web pages.

[0007] Patent document CN109783728B_Page crawler rule update method and system adopts a method based on preset rule updates, which has poor adaptability. Preset rules are generally formulated for specific web page structures or formats. When the HTML structure or layout of the web page changes significantly and new patterns appear that do not meet the rules, the rules may become invalid. Methods based on static rules often cannot capture dynamically generated content (such as content loaded through JavaScript), and require manual addition of new rules, which is time-consuming and labor-intensive.

[0008] Patent document CN117349519A_Web page data collection method device terminal and storage medium, does not specifically explain how to handle the extraction of dynamic pages in the method of extracting HTML documents. The thorny problem that needs to be solved in actual application is how to adapt to the frequent changes of dynamically loaded pages (such as content loaded by JavaScript, dynamically embedded advertisements or verification codes, etc.). Improper handling of this link will lead to incomplete and low-quality collected data samples, affecting the effect of model training; in addition, the invention adopts a feature-based training method to obtain a natural language processing model. The training method is relatively simple and is not universal for websites with different types of structural layouts. The feature rules of different websites are different, and the data training model needs to be reorganized, which requires a large maintenance workload.

[0009] Traditional crawling methods rely on manual adjustment of crawling rules, which is time-consuming and error-prone.

[0010] Therefore, how to efficiently and accurately adapt to changes in web pages for sustainable data collection has become an urgent problem to be solved. Summary of the invention

[0011] The present invention proposes the following technical solutions to address one or more technical deficiencies in the above-mentioned prior art.

[0012] Based on the first aspect of the present invention, a method for adaptively collecting network data based on a pre-trained large model is proposed, comprising:

[0013] S1: Obtain web page data from major website platforms, add HTML tags and custom attributes in the web page data to the vocabulary of the tokenizer, initialize the embedding layer of the pre-trained large model, and use the pre-processed model training samples and the vocabulary to perform incremental training on the pre-trained large model to generate an incremental pre-trained model;

[0014] S2: Generate a large network structure recognition model by performing supervised fine-tuning training on the incremental pre-training model;

[0015] S3: using the web page structure recognition large model to parse the input web page data to obtain web page structured data, and using a difference detection algorithm to compare the web page structured data with the web page standard structure data, generate a structure difference list, and calculate a structure confidence score;

[0016] S4: regularly obtaining screenshots of the target webpage at preset time intervals, performing difference detection on the screenshots obtained successively by using computer vision technology and image comparison algorithm, generating an image difference list, and calculating an image confidence score;

[0017] S5: Combining the structure confidence score and the image confidence score to obtain a comprehensive confidence score, if the comprehensive confidence score is higher than a preset threshold, outputting the web page structured data, generating a specific crawler adjustment strategy according to the structure difference list and the image difference list, and updating the existing crawler rules.

[0018] Furthermore, after obtaining the web page data from the major website platforms, the static web page data and dynamic web page data from the major website platforms are pre-processed, and the web page data are divided into a training set and a validation set as model training samples. The specific steps are:

[0019] S201: In response to the HTML code returned by the http request sent to the target web page, use the HTML parsing library to parse the HTML code and generate a DOM tree;

[0020] S202: traverse the DOM tree and extract the metadata, CSS file links and JavaScript file links of the target webpage to obtain a metadata set, a CSS file link list and a JavaScript file link list of the target webpage;

[0021] S203: Access the target web page through a browser automation tool, capture a screenshot of the target web page after all dynamic contents are loaded, and obtain the content of the DOM tree of the current web page;

[0022] S204: Combine the metadata set of the target web page, the CSS file link list, the JavaScript file link list and the content of the DOM tree of the current web page to construct a target web page model to obtain complete DOM tree structure data of the target web page, and convert it into JSON format.

[0023] Combining static web page and dynamic web page content acquisition together can comprehensively extract all elements of the web page, thus providing a more complete and accurate data foundation.

[0024] Furthermore, the network structure recognition large model is generated by performing supervised fine-tuning training on the incremental pre-training model, which specifically includes: labeling the web page data, generating new data in combination with the large model, constructing web page structure question and answer pairs, and configuring loss functions and evaluation indicators to perform fine-tuning training and iterative optimization on the incremental pre-training model to obtain a large model for web page structure recognition.

[0025] Furthermore, the difference detection algorithm includes a tree structure comparison algorithm, a depth-first search algorithm and a string similarity algorithm. By performing difference detection on the labels, positions and attributes of web page elements, a structural difference list is obtained, and the structural difference list contains the changed web page elements and their changed contents.

[0026] Furthermore, after generating a specific crawler adjustment strategy based on the structure difference list and the image difference list through a rule engine and a strategy generation algorithm, the existing crawler rules are updated according to the crawler adjustment strategy, including modifying the crawler configuration file or updating the selector and extraction logic in the code.

[0027] Combining pre-trained large models, page screenshots and visual comparison can improve the accuracy of web page change detection and data extraction, effectively reduce false detections and missed detections, and improve the robustness and adaptability of the system.

[0028] Furthermore, the specific crawler adjustment strategy includes:

[0029] If the position of a web page element changes, update the crawler's CSS selector or XPath expression;

[0030] If new web page elements appear, add new extraction rules.

[0031] Furthermore, if the crawler adjustment strategy includes redefining the data extraction logic, the data extraction logic is calculated according to the new web page data and the new crawler rules.

[0032] The difference detection algorithm is combined with the crawler strategy to automatically analyze web page changes and adaptively adjust crawler rules, achieving efficient maintenance of the crawler and rapid response to web page updates.

[0033] Based on the second aspect of the present invention, a network data adaptive acquisition system based on a pre-trained large model is also proposed, comprising:

[0034] Initial training module: obtain web page data from major website platforms, add HTML tags and custom attributes in the web page data to the vocabulary of the tokenizer, initialize the embedding layer of the pre-trained large model, and use the pre-processed model training samples and the vocabulary to perform incremental training on the pre-trained large model to generate an incremental pre-trained model;

[0035] Retraining module: generating a large network structure recognition model by performing supervised fine-tuning training on the incremental pre-training model;

[0036] Structural difference module: parses the input web page data using the web page structure recognition model to obtain web page structured data, compares the web page structured data with the web page standard structure data using a difference detection algorithm, generates a structural difference list, and calculates a structural confidence score;

[0037] Image difference module: regularly obtains screenshots of the target web page at pre-set time intervals, performs difference detection on the screenshots obtained successively through computer vision technology and image comparison algorithm, generates an image difference list, and calculates the image confidence score;

[0038] Strategy module: Combine the structure confidence score and the image confidence score to obtain a comprehensive confidence score. If the comprehensive confidence score is higher than a preset threshold, the web page structured data is output. A specific crawler adjustment strategy is generated based on the structure difference list and the image difference list to update the existing crawler rules.

[0039] Based on the third aspect of the present invention, a computer program product is also proposed, comprising one or more computer programs, and when the computer programs are executed by a computer processor, the method described above is implemented.

[0040] The technical effect of the present application is that the present application solves the problem of how to efficiently and accurately adapt to changes in web pages and to sustainably collect web page data. The model can accurately and stably parse the structure of different web pages from a semantic perspective, understand and identify dynamically loaded web page content, and has stronger generalization capabilities. Through automated processing, it reduces the time and cost of manually adjusting crawler rules, can effectively reduce manual intervention, automatically adapt to frequently changing web page structures, improve the efficiency and accuracy of web page data collection, and ensure the continuity and stability of data collection. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Other features, objects and advantages of the present application will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings.

[0042] Figure 1 It is a flowchart of a method for adaptively collecting network data based on a pre-trained large model provided according to an embodiment of the present invention.

[0043] Figure 2 It is a preprocessing flow chart of a network data adaptive collection method based on a pre-trained large model provided according to an embodiment of the present invention.

[0044] Figure 3 It is a framework diagram of a network data adaptive acquisition system based on a pre-trained large model provided according to an embodiment of the present invention.

[0045] Figure 4 The present invention provides a flowchart of adaptive web page collection according to an embodiment of the present invention.

[0046] Figure 5 It is a flow chart of training a large model of web page structure provided according to an embodiment of the present invention.

[0047] Figure 6 It is a schematic diagram of the structure of a computer system suitable for implementing an electronic device of an embodiment of the present invention. DETAILED DESCRIPTION

[0048] The present application will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the relevant invention, rather than to limit the invention. It should also be noted that, for ease of description, only the parts related to the relevant invention are shown in the accompanying drawings.

[0049] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0050] Figure 1A flowchart of a method for adaptively collecting network data based on a pre-trained large model is shown, comprising the following steps:

[0051] S1: Obtain web page data from major website platforms, add HTML tags and custom attributes in the web page data to the vocabulary of the tokenizer, initialize the embedding layer of the pre-trained large model, and use the pre-processed model training samples and the vocabulary to perform incremental training on the pre-trained large model to generate an incremental pre-trained model;

[0052] S2: Generate a large network structure recognition model by performing supervised fine-tuning training on the incremental pre-training model;

[0053] S3: using the web page structure recognition large model to parse the input web page data to obtain web page structured data, and using a difference detection algorithm to compare the web page structured data with the web page standard structure data, generate a structure difference list, and calculate a structure confidence score;

[0054] S4: regularly obtaining screenshots of the target webpage at preset time intervals, performing difference detection on the screenshots obtained successively by using computer vision technology and image comparison algorithm, generating an image difference list, and calculating an image confidence score;

[0055] S5: Combining the structure confidence score and the image confidence score to obtain a comprehensive confidence score, if the comprehensive confidence score is higher than a preset threshold, outputting the web page structured data, generating a specific crawler adjustment strategy according to the structure difference list and the image difference list, and updating the existing crawler rules.

[0056] It should be noted that after obtaining the web page data from various website platforms, the process also includes preprocessing the static web page data and dynamic web page data from various website platforms, and dividing the web page data into a training set and a validation set as model training samples, such as Figure 2 As shown, the specific steps are:

[0057] S201: In response to the HTML code returned by the http request sent to the target web page, use the HTML parsing library to parse the HTML code and generate a DOM tree;

[0058] S202: traverse the DOM tree and extract the metadata, CSS file links and JavaScript file links of the target webpage to obtain a metadata set, a CSS file link list and a JavaScript file link list of the target webpage;

[0059] S203: Access the target web page through a browser automation tool, capture a screenshot of the target web page after all dynamic contents are loaded, and obtain the content of the DOM tree of the current web page;

[0060] S204: Combine the metadata set of the target web page, the CSS file link list, the JavaScript file link list and the content of the DOM tree of the current web page to construct a target web page model to obtain complete DOM tree structure data of the target web page, and convert it into JSON format.

[0061] It should be noted that the generation of a large network structure recognition model by performing supervised fine-tuning training on the incremental pre-training model specifically includes: labeling the web page data, generating new data in combination with the large model, constructing web page structure question-answer pairs, and configuring loss functions and evaluation indicators to perform fine-tuning training and iterative optimization on the incremental pre-training model to obtain a large model for web page structure recognition.

[0062] It should be noted that the difference detection algorithm includes a tree structure comparison algorithm, a depth-first search algorithm and a string similarity algorithm. By performing difference detection on the labels, positions and attributes of web page elements, a structural difference list is obtained, and the structural difference list contains the changed web page elements and their changed contents.

[0063] It should be noted that after generating a specific crawler adjustment strategy based on the structure difference list and the image difference list through the rule engine and the strategy generation algorithm, the existing crawler rules are updated according to the crawler adjustment strategy, including modifying the crawler configuration file or updating the selector and extraction logic in the code.

[0064] It should be noted that the specific crawler adjustment strategy includes:

[0065] If the position of a web page element changes, update the crawler's CSS selector or XPath expression;

[0066] If new web page elements appear, add new extraction rules.

[0067] It should be noted that if the crawler adjustment strategy includes redefining the data extraction logic, the data extraction logic is calculated according to the new web page data and the new crawler rules.

[0068] It should be noted that the web page structure recognition large model is responsible for processing changes at the semantic level and content level, and the web page screenshot and image comparison algorithm compares and processes changes in the web page layout and visual level, and sets different weights according to business needs and actual conditions; if the web page content is updated frequently and the structure is relatively stable, the web page recognition large model is given a higher weight; conversely, if the web page layout changes a lot, the screenshot comparison is given a higher weight; a confidence score is set for each detection result according to the weight, and if the comprehensive confidence score is higher than the threshold, the detection result is adopted.

[0069] It should be noted that the target web page input data includes HTML content, DOM tree, metadata, CSS and JavaScript links, web page dynamic content and page screenshots.

[0070] In a specific embodiment, the browser automation tool used is Selenium, and the HTML parsing library used is BeautifulSoup.

[0071] It should be noted that the pre-trained large model includes the GPT-4 large language model and the Qwen large language model.

[0072] It should be noted that, in the process of incremental pre-training, multiple rounds of training of the pre-trained large model using the expanded vocabulary and pre-processed model training samples can enable the pre-trained large model to better generate and understand texts containing complex HTML structures.

[0073] It should be noted that by annotating web page data from social networking sites and news sites, generating new data in combination with a large model, constructing web page structure question-answer pairs, and configuring loss functions and evaluation indicators to perform fine-tuning training and iterative optimization on the incremental pre-training model, the performance of specific tasks such as web page classification or information extraction can be improved. The large model for web page structure recognition obtained by gradually optimizing the model in a supervised manner can also fully understand and process various types of web page content, and output them in a specified structured format.

[0074] It should be noted that the web page structured data is a standardized format that provides web page related information and classifies web page content, that is, a semantic vocabulary code language. Pages with added web page structured data will have the opportunity to display richer content in the website results page, which will be easier for users to find and thus get more clicks.

[0075] It should be noted that the web page standard structure data is an information set including web page elements, web page element attributes and web page element positions.

[0076] In a specific embodiment, if a new web page element appears, a new extraction rule is added. For example, if a floating advertisement element appears on a website page and covers the extracted web page content, a click event rule is generated to remove the floating advertisement.

[0077] If a web page verification element appears, a simulated real-person click verification will be generated.

[0078] It should be noted that when the layout of a web page changes, the element position or element structure of the website page will change, making the existing XPath or CSS selector invalid, so regular screenshots and difference detection are performed on the target web page.

[0079] It should be noted that updating the existing crawler rules according to the generated specific adjustment strategy can adapt to the new web page structure. If the adjustment strategy involves redefining the data extraction logic, recalculating the data extraction logic according to the new web page structure and the latest crawler rules can ensure the accuracy and completeness of the data extraction.

[0080] It should be noted that the present invention combines static web page and dynamic web page content acquisition together, and can comprehensively extract all elements of the web page, thereby providing a more complete and accurate data basis.

[0081] It should be noted that the present invention combines pre-trained large models, page screenshots and visual comparison to improve the accuracy of web page change detection and data extraction, effectively reduce false detections and missed detections, and improve the robustness and adaptability of the system.

[0082] It should be noted that the present invention combines the difference detection algorithm with the crawler strategy, automatically analyzes the changes in the web page and adaptively adjusts the crawler rules, thereby achieving efficient maintenance of the crawler and quickly responding to web page updates.

[0083] Reference below Figure 3 , which shows a framework diagram of a network data adaptive acquisition system based on a pre-trained large model, including an initial training module a, a retraining module b, a structural difference module c, an image difference module d and a strategy module e.

[0084] In a specific embodiment, the initial training module a is configured to: obtain web page data from major website platforms, add HTML tags and custom attributes in the web page data to the vocabulary of the tokenizer, initialize the embedding layer of the pre-trained large model, and use the pre-processed model training samples and the vocabulary to perform incremental training on the pre-trained large model to generate an incremental pre-trained model.

[0085] In a specific embodiment, the retraining module b is configured to generate a large network structure recognition model by performing supervised fine-tuning training on the incremental pre-training model.

[0086] In a specific embodiment, the structural difference module c is configured to: use the web page structure recognition large model to parse the input web page data to obtain web page structured data, and compare the web page structured data with the web page standard structure data through a difference detection algorithm to generate a structural difference list and calculate a structural confidence score.

[0087] In a specific embodiment, the image difference module d is configured to: regularly obtain screenshots of the target web page at preset time intervals, perform difference detection on the screenshots obtained successively through computer vision technology and image comparison algorithms, generate an image difference list, and calculate an image confidence score.

[0088] In a specific embodiment, the strategy module e is configured to: combine the structure confidence score and the image confidence score to obtain a comprehensive confidence score, output the web page structured data if the comprehensive confidence score is higher than a preset threshold, generate a specific crawler adjustment strategy based on the structure difference list and the image difference list, and update the existing crawler rules.

[0089] It should be noted that the web page adaptive collection process is as follows: Figure 4 As shown, the input data of the target web page is obtained periodically to extract the web page data and compare the web page data structure, including:

[0090] The trained web page structure recognition model is used to parse web page data, and the parsed results are compared with the standard structure of web page data to calculate the structure confidence score;

[0091] Perform visual comparison on the regularly captured webpage screenshots and calculate the image confidence score;

[0092] The comprehensive confidence score is obtained by combining the structure confidence score and the image confidence value score;

[0093] If the comprehensive confidence score is greater than the preset threshold, a crawler adjustment strategy is generated and the crawler rules are updated; otherwise, no adjustment or manual intervention is made to the crawler strategy.

[0094] It should be noted that the training process of the webpage structure model is as follows: Figure 5 As shown, including:

[0095] S101: Load the pre-trained large model;

[0096] S102: Collect various types of web page data;

[0097] S103: Preprocessing web page data, preprocessing static web page data and dynamic web page data in major website platforms, and dividing the web page data into training sets and validation sets as model training samples; specifically including:

[0098] In response to the HTML code returned by the http request sent to the target web page, the HTML code is parsed using an HTML parsing library and a DOM tree is generated;

[0099] Traversing the DOM tree and extracting metadata, CSS file links and JavaScript file links of the target webpage, obtaining a metadata set, a CSS file link list and a JavaScript file link list of the target webpage;

[0100] Access the target page through a browser automation tool, take a screenshot of the target web page after all dynamic content is loaded, and obtain the content of the DOM tree of the current web page;

[0101] Combining the metadata set of the target webpage, the CSS file link list, the JavaScript file link list and the content of the DOM tree of the current webpage, constructing a target webpage model to obtain complete DOM tree structure data of the target webpage, and converting it into JSON format;

[0102] S104: Expand the vocabulary, add the HTML tags and custom attributes in the web page data to the vocabulary of the tokenizer, and initialize the embedding layer of the pre-trained large model;

[0103] S105: performing incremental pre-training on the model using the pre-processed model training samples and the vocabulary to obtain an incremental pre-training model;

[0104] S106: constructing a webpage structure question-answer pair;

[0105] S107: Performing supervised fine-tuning training on the incremental pre-training model, configuring a loss function and an evaluation index to perform fine-tuning training and iterative optimization on the incremental pre-training model;

[0106] S108: Obtain a large model of the web page structure.

[0107] This application solves the problem of how to efficiently and accurately adapt to changes in web pages and to sustainably collect web page data. The model can accurately and stably parse the structure of different web pages from a semantic perspective, understand and identify dynamically loaded web page content, and has stronger generalization capabilities. Through automated processing, it reduces the time and cost of manually adjusting crawler rules, can effectively reduce manual intervention, automatically adapt to frequently changing web page structures, improve the efficiency and accuracy of web page data collection, and ensure the continuity and stability of data collection.

[0108] Reference below Figure 6 , which shows a schematic diagram of the structure of a computer system suitable for implementing an electronic device of an embodiment of the present application. Figure 6 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0109] like Figure 6 As shown, the computer system includes a central processing unit (CPU) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage part 608 into a random access memory (RAM) 603. Various programs and data required for system operation are also stored in the RAM 603. The CPU 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0110] The following components are connected to the I / O interface 605: an input section 606 including a keyboard, a mouse, etc.; an output section 607 including a liquid crystal display (LCD), etc. and a speaker, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN card, a modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I / O interface 605 as needed. A removable medium 611, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 610 as needed, so that a computer program read therefrom is installed into the storage section 608 as needed.

[0111] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable storage medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 609, and / or installed from the removable medium 611. When the computer program is executed by the central processing unit (CPU) 601, the above functions defined in the method of the present application are executed. It should be noted that the computer-readable storage medium of the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium can be, for example, - but not limited to - an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection with one or more conductors, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device, or device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable storage medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, device, or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wireless, wireline, optical cable, RF, etc., or any suitable combination of the foregoing.

[0112] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0113] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0114] The modules involved in the embodiments of the present application may be implemented by software or by hardware.

[0115] As another aspect, the present application also provides a computer-readable storage medium, which may be included in the electronic device described in the above embodiment; or it may exist independently without being assembled into the electronic device. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed by the electronic device, the electronic device: obtains web page data from major website platforms, adds HTML tags and custom attributes in the web page data to the tokenizer Token izer's vocabulary, initialize the embedding layer of the pre-trained large model, and use the pre-processed model training samples and the vocabulary to perform incremental training on the pre-trained large model to generate an incremental pre-trained model; generate a network structure recognition large model by performing supervised fine-tuning training on the incremental pre-trained model; use the web page structure recognition large model to parse the input web page data to obtain web page structured data, and use the difference detection algorithm to compare the web page structured data with the web page standard structure data to generate a structure difference list and calculate the structure confidence score; regularly obtain screenshots of the target web page at preset time intervals, use computer vision technology and image comparison algorithm to perform difference detection on the screenshots obtained successively, generate an image difference list, and calculate the image confidence score; combine the structure confidence score and the image confidence score to obtain a comprehensive confidence score, if the comprehensive confidence score is higher than a preset threshold, output the web page structured data, generate a specific crawler adjustment strategy according to the structure difference list and the image difference list, and update the existing crawler rules.

[0116] The above description is only a preferred embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the present application is not limited to the technical solution formed by a specific combination of the above-mentioned technical features, but should also cover other technical solutions formed by any combination of the above-mentioned technical features or their equivalent features without departing from the above-mentioned inventive concept. For example, the above-mentioned features are replaced with the technical features with similar functions disclosed in this application (but not limited to) to form a technical solution. Finally, it should be explained that the above embodiments are only used to illustrate but not limit the technical solutions of the present invention. Although the present invention is described in detail with reference to the above embodiments, those of ordinary skill in the art should understand that the present invention can still be modified or replaced by equivalents, and any modification or partial replacement that does not depart from the spirit and scope of the present invention should be included in the scope of the claims of the present invention.

Claims

1. A network data adaptive acquisition method based on a pre-trained large model, characterized in that: include: S1: Obtain web page data from major website platforms, add HTML tags and custom attributes in the web page data to the vocabulary of the tokenizer, initialize the embedding layer of the pre-trained large model, and use the pre-processed model training samples and the vocabulary to perform incremental training on the pre-trained large model to generate an incremental pre-trained model; S2: Generate a large network structure recognition model by performing supervised fine-tuning training on the incremental pre-training model; S3: using the web page structure recognition large model to parse the input web page data to obtain web page structured data, and using a difference detection algorithm to compare the web page structured data with the web page standard structure data, generate a structure difference list, and calculate a structure confidence score; S4: regularly obtaining screenshots of the target webpage at preset time intervals, performing difference detection on the screenshots obtained successively by using computer vision technology and image comparison algorithm, generating an image difference list, and calculating an image confidence score; S5: Combining the structure confidence score and the image confidence score to obtain a comprehensive confidence score, if the comprehensive confidence score is higher than a preset threshold, outputting the web page structured data, generating a specific crawler adjustment strategy according to the structure difference list and the image difference list, and updating the existing crawler rules.

2. The method according to claim 1, characterized in that After obtaining the web page data from major website platforms, the process also includes preprocessing the static web page data and dynamic web page data from the major website platforms, dividing the web page data into a training set and a validation set and using them as model training samples. The specific steps are: S201: In response to the HTML code returned by the http request sent to the target web page, use the HTML parsing library to parse the HTML code and generate a DOM tree; S202: traversing the DOM tree and extracting metadata, CSS file links and JavaScript file links of the target webpage to obtain a metadata set, a CSS file link list and a JavaScript file link list of the target webpage; S203: Access the target web page through a browser automation tool, capture a screenshot of the target web page after all dynamic contents are loaded, and obtain the content of the DOM tree of the current web page; S204: Combine the metadata set of the target web page, the CSS file link list, the JavaScript file link list and the content of the DOM tree of the current web page to construct a target web page model to obtain complete DOM tree structure data of the target web page, and convert it into JSON format.

3. The method according to claim 1, characterized in that Generating a large model for network structure recognition by performing supervised fine-tuning training on the incremental pre-training model specifically includes: labeling the web page data, generating new data in combination with the large model, constructing web page structure question-answer pairs and configuring loss functions and evaluation indicators to perform fine-tuning training and iterative optimization on the incremental pre-training model to obtain a large model for web page structure recognition.

4. The method according to claim 1, characterized in that: The difference detection algorithm includes a tree structure comparison algorithm, a depth-first search algorithm and a string similarity algorithm. By performing difference detection on the labels, positions and attributes of web page elements, a structural difference list is obtained. The structural difference list contains the changed web page elements and their changed contents.

5. The method according to claim 1, characterized in that: According to the structure difference list and the image difference list, a specific crawler adjustment strategy is generated through a rule engine and a strategy generation algorithm, and then the existing crawler rules are updated according to the crawler adjustment strategy, including modifying the crawler configuration file or updating the selector and extraction logic in the code.

6. The method according to claim 5, characterized in that The specific crawler adjustment strategies include: If the position of a web page element changes, update the crawler's CSS selector or XPath expression; If new web page elements appear, add new extraction rules.

7. The method according to claim 5, characterized in that If the crawler adjustment strategy includes redefining the data extraction logic, the data extraction logic is calculated according to the new web page data and the new crawler rules.

8. A network data adaptive acquisition system based on a pre-trained large model, characterized in that: include: Initial training module: obtain web page data from major website platforms, add HTML tags and custom attributes in the web page data to the vocabulary of the tokenizer, initialize the embedding layer of the pre-trained large model, and use the pre-processed model training samples and the vocabulary to perform incremental training on the pre-trained large model to generate an incremental pre-trained model; Retraining module: generating a large network structure recognition model by performing supervised fine-tuning training on the incremental pre-training model; Structural difference module: parses the input web page data using the web page structure recognition model to obtain web page structured data, compares the web page structured data with the web page standard structure data using a difference detection algorithm, generates a structural difference list, and calculates a structural confidence score; Image difference module: regularly obtains screenshots of the target web page at pre-set time intervals, performs difference detection on the screenshots obtained successively through computer vision technology and image comparison algorithm, generates an image difference list, and calculates the image confidence score; Strategy module: Combine the structure confidence score and the image confidence score to obtain a comprehensive confidence score. If the comprehensive confidence score is higher than a preset threshold, the web page structured data is output. A specific crawler adjustment strategy is generated according to the structure difference list and the image difference list to update the existing crawler rules.

9. A computer program product comprising one or more computer programs, characterized in that When the computer program is executed by a computer processor, the method according to claims 1 to 7 is implemented.

Images