Method and system for automatically extracting news detail page XPath based on large model

Abstract

The invention belongs to the technical field of computers, and discloses a news detail page XPath automatic extraction method and system based on a large model, and the method comprises the steps: determining a large model base through multi-aspect evaluation; the method comprises the following steps: preprocessing an HTML page, designing a structured prompt template and constructing a thinking chain CoT data set; utilizing a Lora technology fine tuning model, and adopting a smooth loss function and a dual reward mechanism of fusion format and content evaluation to optimize; and performing format calibration on the XPath and the JSON output by the model, and extracting and verifying the validity of the URL of the detail page by using the calibrated XPath. Compared with a traditional template-based information extraction method, the method does not depend on the stability of a webpage structure, has higher generalization ability and fault tolerance, remarkably improves the information extraction precision, enhances the complex webpage adaptability, improves the overall robustness and universality of a system, and is suitable for large-scale popularization and application. And a stable extraction effect can still be kept in news websites with frequent structure change.

Description

Technical Field

[0001] This invention belongs to the field of computer technology, and in particular relates to a method and system for automatic XPath extraction of news detail pages based on a large model. Background Technology

[0002] Most current mainstream web page information extraction methods revolve around web page structural features or visual presentation features, such as common structural template matching, content density analysis, and some traditional machine learning classification methods. These methods can achieve certain results when processing static, well-structured pages, but when faced with news media websites that are structurally complex, have diverse content, or are frequently updated, they often exhibit problems such as insufficient versatility, poor adaptability, and high maintenance costs.

[0003] In recent years, with the rapid development of large language models, the text understanding and information extraction capabilities based on pre-trained models have been significantly improved, providing a new solution for web page information extraction tasks. However, existing general-purpose large models still suffer from problems such as field misalignment, content omissions, and strong randomness in extraction results in terms of structured output, making it difficult to directly meet the high-precision and diverse field extraction requirements of news web pages. Furthermore, optimizations for web page structure and specific task requirements are still insufficient, lacking task-specific model fine-tuning mechanisms and instruction design strategies, which affects the stability and controllability of practical application results.

[0004] Based on the above analysis, the problems and shortcomings of the existing technology are as follows:

[0005] (1) Existing web page information extraction methods often exhibit problems such as insufficient universality, poor adaptability, and high maintenance costs when faced with news media websites that are complex in structure, diverse in content, or frequently updated.

[0006] (2) Existing general large models have problems such as field disorder, content omission and strong randomness in structured output, which make it difficult to directly meet the high-precision and diversified field extraction needs of news web pages.

[0007] (3) The optimization of web page structure and specific task requirements is insufficient, and there is a lack of model fine-tuning mechanism and instruction design strategy for specific tasks, which affects the stability and controllability of actual application effect. Summary of the Invention

[0008] To overcome the problems of low accuracy in webpage information extraction, unstable structured output results, poor model adaptability, and low efficiency in existing technologies, this invention discloses an automatic XPath extraction method and system for news detail pages based on a large model, particularly relating to an automatic generation method and system for news media list page HTML detail page XPath extraction based on a large model. The technical solution is as follows:

[0009] This invention is implemented as follows: A method for automatically extracting XPath from news detail pages based on a large model includes the following steps:

[0010] Model evaluation steps: Evaluate multiple large models by assessing their multilingual recognition capabilities, long text reasoning capabilities, code execution capabilities, and resource consumption to determine the optimal large model base;

[0011] Steps for building a multilingual dataset: Collect list page HTML pages from multiple news websites, covering multiple languages; locate elements, clean and trim the data of the HTML pages; design structured prompt templates and build the Mind Chain CoT dataset; divide the dataset into training set and test set.

[0012] Adaptive learning steps: Fine-tune the selected best large model base using LoRa technology, set fine-tuning parameters, optimize the model using a smooth loss function until convergence, and introduce a dual reward mechanism for training; the dual reward mechanism includes format reward and content reward, the format reward is based on the correctness of the output format, and the content reward is based on the accuracy of the extracted content;

[0013] Format calibration step: Use a format calibration tool to correct the XPath expression and JSON format of the model output to ensure that the output conforms to the expected structure;

[0014] Furthermore, the format calibration tools include: an input module, an XPath calibration module, a JSON calibration module, and an output module;

[0015] Input module: Responsible for receiving the raw JSON content output by the model. First, it checks whether the input is in the correct JSON format, provides standardized input for the XPath calibration module, and uses regular expressions to extract the array content corresponding to the news_url_xpath field in the raw content and store it into the "XPath list".

[0016] XPath calibration module: 1. Path symbol verification: Checks whether it contains key fields such as " / / ", " / ", "@", and "href"; 2. Keyword logic verification: Verifies whether "@href" is valid. It needs to be bound to the tag. If there is only "@href", it is invalid. For example, " / / h3[@class="entry-title"] / a / @href" is correct, while " / / a / @href" is invalid because it contains many invalid links.

[0017] JSON calibration module: Performs standardization validation on the overall JSON format of the original input; 1. Symbol standardization: Checks whether it starts with "{" and ends with "}", and whether they match, and whether it contains structural symbols such as ":" key-value separator and "," element separator; 2. Content standardization: Checks whether there are no extra escape characters "\" in the XPath expression.

[0018] Output module: Integrates the verified content and outputs the corrected XPath list and calibration log, making it easy to directly observe which content has been modified.

[0019] XPath Application and Validation Steps: Apply the formatted XPath expression to the HTML source code of the news list page, extract the details page URL, and verify the validity and accuracy of the URL.

[0020] Specifically, the calibrated XPath is applied to the target webpage, and the webpage response of the target list page is obtained through an HTTP request. The corresponding href attribute value in the HTML source code of the response body is extracted, and the extracted content is checked to see if it is consistent with the original webpage content and if the format is correct. Invalid links are removed, and the number of actual detail page links on the webpage is compared with the number of links extracted by XPath to calculate the pass rate.

[0021] In the model evaluation step, the optimal large model base is the Qwen3-Coder-30B-A3B-Instruct model, which serves as the basic model for knowledge extraction.

[0022] In the multilingual dataset construction step, the data cleaning includes removing invalid elements, noisy elements, and transforming label elements.

[0023] In the multilingual dataset construction step, the structured prompt template is used to guide the model to parse HTML pages and extract key content, outputting list page titles and URL information in structured JSON format.

[0024] In the adaptive learning step, the LoRa fine-tuning is achieved by adding a low-rank trainable parameter matrix, the fine-tuning parameters including the learning rate, the number of training epochs, and gradient accumulation.

[0025] In the adaptive learning step, the Smoothed loss function is used to optimize the model and train until the loss function converges; the smoothed loss function is the smoothed L1 loss function, which modifies the calculation method of the original label or loss value by introducing a smoothing coefficient or correction factor.

[0026]

[0027] Where x is the difference between the predicted value and the true value; the smooth L1 loss treats smaller errors as squared errors, while for larger errors, it uses the L1 loss.

[0028] In the adaptive learning step, the format reward in the dual reward mechanism includes:

[0029] The JSON format generated by the input model and the XPath format generated by the model;

[0030] Check the syntax correctness, field integrity, and XPath validity of the JSON through a format verification tool;

[0031] Output a score between 0 and 3, with 3 points when it is completely correct;

[0032] The calculation formula for the format reward is:

[0033] R_format = 1.5×(JSON syntax correctness) + 1.5×(XPath format validity)

[0034] The content reward includes:

[0035] The content generated by the input model, including the XPath and URL extraction results of the list page;

[0036] Check whether the extracted URL is consistent with the actual page URL through the content evaluation module, verify the validity of the URL extracted by the model, ensure that the extracted URL is valid and points to the correct page; for each URL extracted by the model, check whether it is consistent with the actual page URL, if consistent, it is recorded as a matching item; count the number of URLs extracted by the XPath given by the model that match the actual page URL;

[0037] When a total of N URLs are extracted, and M of them match correctly, it is specifically divided into two cases:

[0038] When N < M, the URL matching degree is full marks; when N > M, the URL matching degree = M / N;

[0039] Output a score between 0 and 7, with 7 points when it is completely consistent;

[0040] The calculation formula for the content reward is:

[0041] R_content = 7×(URL matching degree)

[0042] The total reward is the sum of the format reward and the content reward, and the calculation formula for the total reward is:

[0043] R_total = R_format + R_content.

[0044] In the format calibration step, for rules that extract list page links from large model recognition XPath, the format calibration tool retains only valid URL links; in the case of incorrect format, the format calibration tool corrects the incorrect format to ensure that it meets the expected structure and validity.

[0045] For the model output, verify whether the XPath expression extracted by the model meets the target format and structure requirements, and verify whether it accurately extracts valid title links from the news list page;

[0046] Check whether the data generated by the large model is in standard JSON format, contains the correct field structure, and ensure the integrity and validity of the data;

[0047] Check if the generated XPath expression is formatted correctly, verify that the XPath expression contains the required element selectors, and ensure that there are no omissions or syntax errors. Extract valid detail page URL links.

[0048] In the XPath application and verification steps, the calibrated XPath is applied to the HTML source code through an automated script, and the extracted URL is verified through an HTTP request to see if it can return a valid news details page.

[0049] Another objective of this invention is to provide an automatic XPath extraction system for news detail pages based on a large model. This system is used to implement the aforementioned automatic XPath extraction method for news detail pages based on a large model. The system includes:

[0050] The model evaluation module is used to evaluate multiple large models based on multilingual recognition capabilities, long text reasoning capabilities, code execution capabilities, and resource consumption, and to determine the best large model base.

[0051] The multilingual dataset construction module is used to collect list page HTML pages from multiple news websites, covering multiple languages. It performs element location, data cleaning and cropping on the HTML pages, designs structured prompt templates and builds the Mind Chain CoT dataset, and divides the dataset into training set and test set.

[0052] The adaptive learning module is used to fine-tune the selected best large model base using LoRa technology, set fine-tuning parameters, optimize the model using a smooth loss function until convergence, and introduce a dual reward mechanism for training, which includes format reward and content reward.

[0053] The format calibration module is used to correct the XPath expressions and JSON format of the model output using a format calibration tool to ensure that the output conforms to the expected structure.

[0054] The XPath application and validation module is used to apply formatted XPath expressions to the HTML source code of the news list page, extract the details page URL, and validate the validity and accuracy of the URL.

[0055] Combining all the above technical solutions, the beneficial effects of this invention are as follows:

[0056] First, this invention significantly improves information extraction accuracy. By constructing a dedicated XPath extraction method for structured text on news web pages, it effectively enhances the accuracy of information extraction on news web pages with different structures and formats. Second, this invention enhances adaptability to complex web pages. By introducing a large model base and supervised fine-tuning strategies, this invention can adapt to various web page structures, maintaining stable extraction performance, especially in complex pages with dynamic loading and nested structures. Third, this invention improves the overall robustness and versatility of the system: by combining prompt word engineering with training on diverse datasets, the model possesses strong transfer capabilities and generalization performance across different media websites and domain scenarios, making it more applicable. In this way, the system can automatically generate and validate XPaths, thereby significantly reducing the time and cost of manual parsing, effectively saving manpower and improving work efficiency.

[0057] Secondly, compared with traditional template-based information extraction methods, the XPath automatic extraction method for news detail pages based on large models in this invention does not rely on the stability of the webpage structure, has stronger generalization ability and fault tolerance, and can still maintain stable extraction results in news websites with frequent structural changes.

[0058] Third, traditional manual XPath extraction is time-consuming, and different websites require different XPath codes. When the webpage structure changes, the code needs to be rewritten. This invention uses a large model to automatically generate, calibrate, and verify a fully automated process, reducing the time to extract XPath links from a single website's detail page by less than 20 seconds and lowering labor costs by over 80%. Furthermore, for clients with needs such as multi-module news platforms and public opinion monitoring, it can quickly cover thousands of websites, significantly improving data collection efficiency. Currently, there are two main extraction technologies both domestically and internationally: one is based on traditional templates, which has poor extraction results for multiple different websites and weak generalization ability; the other directly uses large models for extraction, lacking format verification and validation, resulting in unstable extraction accuracy. This invention corrects format issues through a calibration module, providing a more reliable technical path for structured webpage extraction. Because website structures can vary, leading to incomplete content extraction, this invention trains and fine-tunes the model using diverse data, allowing it to learn the XPath generation rules for different webpages, reducing basic syntax errors, and automatically adapting to webpage structures and stably extracting content information without manual intervention. Since large models have powerful text generation capabilities, most technical solutions assume that the XPath output by large models can be used directly. However, this invention proves through a format calibration module that the generated results of large models contain some non-standard or invalid content, so professional calibration is required to meet the needs of practical applications. Attached Figure Description

[0059] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure;

[0060] Figure 1 This is a flowchart of the method for automatic XPath extraction of news detail pages based on a large model provided in an embodiment of the present invention;

[0061] Figure 2 This is a flowchart of the text structured XPath extraction method based on a large model provided in this embodiment of the invention;

[0062] Figure 3 This is a flowchart of the text structured XPath information extraction and processing provided in the embodiments of the present invention. Detailed Implementation

[0063] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Many specific details are set forth in the following description to provide a thorough understanding of the present invention. However, the present invention can be practiced in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0064] The innovations of this invention are as follows: A task-oriented Prompt template is designed to improve extraction accuracy and address the issue of deviation from the target; LoRa low-rank fine-tuning is added to the large model to reduce resource consumption and enhance adaptability; a dual reward mechanism is used to improve training performance and address the problem of incomplete evaluation; and format calibration technology is added at the final output to resolve the issue of format errors affecting the overall extraction. This results in an increase in XPath extraction accuracy of the large model on multilingual websites from 71.7% to 98.1%.

[0065] Specifically as follows:

[0066] 1. Task-oriented Prompt Template Design: A structured prompt template was designed for two extraction tasks: HTML parsing and detail page XPath extraction. The task execution steps and expected output format were clearly defined. The output of one XPath was improved to five XPaths, resulting in more comprehensive coverage. The extraction accuracy increased from 85.3% to 99.9%, effectively extracting all target links from the webpage.

[0067] 2. Low-rank parameter fine-tuning method based on LoRa technology: By attaching a low-rank trainable parameter matrix, efficient fine-tuning of pre-trained large models can be achieved, significantly reducing the size of training parameters and resource consumption, and improving the model's extraction accuracy and robustness under different news webpage structures.

[0068] 3. Dual Reward Mechanism: Enhancing the traditional GRPO reward system's format reward and answer reward:

[0069] Formatting Bonus (0-3 points): Based on the original format, evaluate whether the output format of the extracted XPath results is correct;

[0070] Content Bonus (0-7 points): For the extraction results, evaluate whether the extracted content is consistent with the standard answer. The evaluation criterion is accuracy.

[0071] The cumulative reward varies between 0 and 10.

[0072] 4. Format calibration: In cases of incorrect formatting, the format calibration tool can be used to correct the errors in order to facilitate content evaluation.

[0073] Example 1, such as Figures 1-3 As shown, the automatic XPath extraction method for news detail pages based on large models provided in this embodiment of the invention includes the following steps:

[0074] S1, Model Evaluation: Perform performance evaluation on multiple models to ensure that the base model performs as required in different scenarios.

[0075] First, the most suitable large model base was selected for extracting information from news media web pages. The model was evaluated from multiple aspects, including multilingual recognition ability, long text reasoning ability, code execution ability, and resource consumption. Finally, the Qwen3-Coder-30B-A3B-Instruct model was selected as the basic model for knowledge extraction.

[0076] S2, Multilingual Dataset Construction: A total of 3460 news webpages (HTML pages) were collected from 41 news websites, covering multiple languages ​​including Chinese, English, Russian, German, Arabic, and Japanese. Element location, data cleaning, and trimming were performed on the webpage data, including removing invalid and noisy elements and transforming tag elements. Inspired by "A Machine Learning-Based Method for Identifying Harmful Information on Webpages," this invention designs a Prompt template, constructs a CoT (Coding on Thought) prompt library, and completes the division of the training and test sets, specifically including:

[0077] (1) The data was collected from HTML pages of 41 news websites (such as CBS News, Voice of Turkey, People's Daily Online, etc.), covering multiple languages ​​including English, Chinese, and Russian. The specific proportions are shown in Table 2. The collected data needs to be cleaned, and invalid and noisy elements need to be removed to obtain valuable data.

[0078] (2) Design specific prompt templates to guide the model in parsing HTML pages and extracting key content. This involves the accurate location and extraction of information such as list page titles and URLs, while leveraging the reasoning capabilities of large models to handle the diversity of different webpage structures and languages. The extraction results are output in a structured JSON format to ensure efficient and accurate information extraction, applicable to various types of news websites. A CoT (Conceptual Chain) dataset is constructed based on the characteristics of different websites and language types. Finally, the dataset is divided into training and test sets in a 9:1 ratio. The extraction process for news list page titles and URLs is as follows:

[0079] First, the page structure is processed using an HTML parsing library to locate the parent container containing multiple items (such as '...'). 'or' Then, label recognition is performed to find duplicate entry units (such as '). 'or' <article>Then search within the entry for the title text and its associated '); The code snippet describes a process for extracting target data from XPath expressions. It filters valid paths (e.g., those containing ' / news / ' or '.html') and excludes ad or script links. Finally, it extracts the target data using concise XPath expressions (level ≤ 3, ending with '@href') and outputs the XPath expressions as structured JSON.

[0080] S3, Adaptive Learning: Based on the LoRa principle, fine-tuning the parameters of a large model; setting fine-tuning parameters (such as learning rate, number of training epochs, gradient accumulation, etc.); using a loss function to optimize the model until convergence, ensuring that the model achieves an accuracy and F1 score of over 90% on the task dataset; and employing a dual reward mechanism, enhancing the format reward and answer reward of the traditional GRPO reward, specifically including:

[0081] (1) Using the LoRa fine-tuning principle to fine-tune the model can effectively reduce training parameters and improve training efficiency.

[0082] (2) The model is optimized using the Smoothed loss function and trained until the loss function converges. The Smoothed loss function reduces overfitting to outliers and enhances generalization ability by smoothing the loss function. The smoothed loss function typically modifies the calculation of the original labels or loss values ​​by introducing a smoothing coefficient or correction factor. The specific formula is as follows:

[0083]

[0084] Here, x represents the difference between the predicted and true values. The smoothed L1 loss treats smaller errors as squared errors, while using L1 loss for larger errors. This invention addresses the problem of overfitting and weak generalization ability in news list page information extraction models due to the diversity of webpage structures and noise interference during training. Therefore, it innovatively introduces the Smoothed loss function as a model optimizer.

[0085] (3) A dual reward mechanism is adopted, and the specific steps are as follows:

[0086] 1) Format reward:

[0087] Input: JSON format generated by the model, XPath format generated by the model;

[0088] Processing: Use format validation tools to check the JSON syntax, field completeness, and XPath validity;

[0089] Output: A score between 0 and 3, with 3 points for a perfect score.

[0090] Calculation formula:

[0091] R_format = 1.5×(JSON syntax correctness) + 1.5×(XPath format validity)

[0092] Among them, checking the JSON syntax correctness, field integrity, XPath validity, etc. through the format verification tool includes: First, verify the syntax correctness (no basic errors such as unmatched parentheses or missing quotes); then, verify the field integrity according to the preset field definitions, confirm that the required field "XPath" is included and the type conforms to the specification; finally, execute the XPath expression in the original HTML page to verify whether they can effectively and accurately locate and extract the target content.

[0093] 2) Content reward:

[0094] Input: The content generated by the model (the XPath and URL extraction results of the list page)

[0095] Processing: Check whether the extracted URL is consistent with the URL of the actual page through the content evaluation module, verify the validity of the URL extracted by the model, and ensure that the extracted URL is valid and can point to the correct page. For each URL extracted by the model, check whether it is consistent with the URL of the actual page. If it is consistent, it is recorded as a matching item. Count the number of URLs extracted by the XPath given by the model that match the actual page URL. Suppose a total of N URLs are extracted, and M of them match correctly. Specifically, it is divided into two cases: ① When N < M, the URL matching degree is full marks; ② When N > M, the URL matching degree = M / N.

[0096] Output: A score between 0 and 7, 7 points when it is completely consistent.

[0097] Calculation formula:

[0098] R_content = 7×(URL matching degree)

[0099] 3) Total reward calculation:

[0100] R_total = R_format + R_content.

[0101] S4, Format calibration: For the rules for the large model to identify and generate the XPath of the list page link, only retain the valid URL links through the format calibration tool.

[0102] In the case of incorrect format, correct the incorrect format through the format calibration tool to ensure compliance with the expected structure and validity.

[0103] Output content validation: Verify whether the XPath expression extracted by the model conforms to the target format and structure requirements. Verify whether it can accurately extract valid title links from the news list page.

[0104] (1) Validate JSON format: Check whether the data generated by the large model is in standard JSON format, contains the correct field structure, and ensures the integrity and validity of the data.

[0105] (2) Validate XPath format: Check if the generated XPath expression is formatted correctly. This mainly verifies whether the XPath expression contains the necessary element selectors, such as @href, and ensures there are no omissions or syntax errors, so that a valid details page URL can be extracted.

[0106] Example 2: The XPath automatic extraction system for news detail pages based on a large model provided in this embodiment of the invention includes:

[0107] The model evaluation module is used to evaluate multiple large models based on multilingual recognition capabilities, long text reasoning capabilities, code execution capabilities, and resource consumption, and to determine the best large model base.

[0108] The multilingual dataset construction module is used to collect list page HTML pages from multiple news websites, covering multiple languages. It performs element location, data cleaning and cropping on the HTML pages, designs structured prompt templates and builds the Mind Chain CoT dataset, and divides the dataset into training set and test set.

[0109] The adaptive learning module is used to fine-tune the selected best large model base using LoRa technology, set fine-tuning parameters, optimize the model using a smooth loss function until convergence, and introduce a dual reward mechanism for training, which includes format reward and content reward.

[0110] The format calibration module is used to correct the XPath expressions and JSON format of the model output using a format calibration tool to ensure that the output conforms to the expected structure.

[0111] The XPath application and validation module is used to apply formatted XPath expressions to the HTML source code of the news list page, extract the details page URL, and validate the validity and accuracy of the URL.

[0112] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0113] To further demonstrate the positive effects of the above embodiments, the present invention conducts the following model comparison experiment based on the above technical solution.

[0114] Accuracy comparisons clearly show that the improved model performs better on websites in different languages. Before fine-tuning, the average accuracy across seven languages ​​was approximately 71.7%, with the lowest accuracy at 59.7% for Japanese websites. After fine-tuning, the average accuracy reached 98.1%, with 100% accuracy achieved for six languages: Traditional Chinese, Turkish, Russian, Simplified Chinese, English, and Japanese. The multilingual website recognition capability is significantly improved, as detailed in Table 1. Dataset details are shown in Table 2.

[0115] Table 1 Comparison of URL extraction accuracy for news detail pages on websites in different languages

[0116]

[0117] Table 2. Detailed information about the dataset.

[0118] Application Example 1: Test Link: https: / / www.ifeng.com /

[0119] Large model outputs XPath content:

[0120]

[0121] The output content after correction:

[0122]

[0123] Application Example 2: Test Link: https: / / www.tokyo-np.co.jp / Large Model Output XPath Content:

[0124]

[0125] The output content after correction:

[0126]

[0127] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications, equivalent substitutions and improvements made by those skilled in the art within the scope of the technology disclosed in the present invention and within the spirit and principles of the present invention should be covered within the scope of protection of the present invention. < / article>

Claims

1. A method for automatic XPath extraction of news detail pages based on a large model, characterized in that, include: Model evaluation steps: Evaluate multiple large models by assessing their multilingual recognition capabilities, long text reasoning capabilities, code execution capabilities, and resource consumption to determine the optimal large model base; Steps for building a multilingual dataset: Collect list page HTML pages from multiple news websites, covering multiple languages; locate elements, clean and trim the data of the HTML pages; design structured prompt templates and build the Mind Chain CoT dataset; divide the dataset into training set and test set. Adaptive learning steps: Fine-tune the selected best large model base using LoRa technology, set fine-tuning parameters, optimize the model using a smooth loss function until convergence, and introduce a dual reward mechanism for training; the dual reward mechanism includes format reward and content reward, the format reward is based on the correctness of the output format, and the content reward is based on the accuracy of the extracted content; Format calibration step: Use a format calibration tool to correct the XPath expression and JSON format of the model output to ensure that the output conforms to the expected structure; XPath Application and Validation Steps: Apply the formatted XPath expression to the HTML source code of the news list page, extract the details page URL, and verify the validity and accuracy of the URL.

2. The method for automatic XPath extraction of news detail pages based on a large model according to claim 1, characterized in that, In the model evaluation step, the optimal large model base is the Qwen3-Coder-30B-A3B-Instruct model, which serves as the basic model for knowledge extraction.

3. The method for automatic XPath extraction of news detail pages based on a large model according to claim 1, characterized in that, In the multilingual dataset construction step, the data cleaning includes removing invalid elements, noisy elements, and transforming label elements.

4. The method for automatic XPath extraction of news detail pages based on a large model according to claim 1, characterized in that, In the multilingual dataset construction step, the structured prompt template is used to guide the model to parse HTML pages and extract key content, outputting list page titles and URL information in structured JSON format.

5. The method for automatic XPath extraction of news detail pages based on a large model according to claim 1, characterized in that, In the adaptive learning step, the LoRa fine-tuning is achieved by adding a low-rank trainable parameter matrix, the fine-tuning parameters including the learning rate, the number of training epochs, and gradient accumulation.

6. The method for automatic XPath extraction of news detail pages based on a large model according to claim 1, characterized in that, In the adaptive learning step, the Smoothed loss function is used to optimize the model and train until the loss function converges; the smoothed loss function is the smoothed L1 loss function, which modifies the calculation method of the original label or loss value by introducing a smoothing coefficient or correction factor. Here, x is the difference between the predicted value and the true value; the smoothing L1 loss treats smaller errors as squared errors, while for larger errors, the L1 loss is used.

7. The method for automatic XPath extraction of news detail pages based on a large model according to claim 1, characterized in that, In the adaptive learning step, the formatted reward in the dual reward mechanism includes: Input the JSON format generated by the model and the XPath format generated by the model; Use format validation tools to check the JSON syntax, field integrity, and XPath validity; Output a score between 0 and 3, with 3 points for a completely correct answer; The formula for calculating the reward in the specified format is as follows: R_format = 1.5 × (JSON syntax correctness) + 1.5 × (XPath format validity) The content rewards include: The input model generates content, including the XPath and URL extraction results of the list page; Check whether the extracted URL is consistent with the URL of the actual page through the content evaluation module, verify the validity of the URL extracted by the model, ensure that the extracted URL is valid and points to the correct page; for each URL extracted by the model, check whether it is consistent with the URL of the actual page. If it is consistent, it is recorded as a matching item; count the number of URLs extracted by the XPath given by the model that match the actual page URL. When a total of N URLs are extracted, and M of them match correctly, it is specifically divided into two cases: When N < M, the URL matching degree is full marks; when N > M, the URL matching degree = M / N; Output a score between 0 and 7, and it is 7 points when it is completely consistent; The calculation formula for the content reward is as follows: R_content = 7×(URL matching degree) The total reward is the sum of the format reward and the content reward. The calculation formula for the total reward is as follows: R_total = R_format + R_content.

8. The method for automatic XPath extraction of news detail pages based on a large model according to claim 1, characterized in that, In the format calibration step, for the rules for the large model to identify and generate the XPath for extracting the list page link, only the valid URL links are retained through the format calibration tool; in the case of incorrect format, the incorrect format is corrected through the format calibration tool to ensure compliance with the expected structure and validity; For the content output by the model, verify whether the XPath expression extracted by the model meets the target format and structure requirements, and verify whether the valid title links in the news list page are accurately extracted; Check whether the data generated by the large model is in the standard JSON format, contains the correct field structure, and ensure the integrity and validity of the data; Check whether the format of the generated XPath expression is correct, verify whether the XPath expression contains the required element selectors, and ensure that there are no omissions or syntax errors, and extract the valid detail page URL links.

9. The method for automatic XPath extraction of news detail pages based on a large model according to claim 1, characterized in that, In the XPath application and verification step, apply the calibrated XPath to the HTML source code through an automated script, and verify whether the extracted URL can return a valid news detail page through an HTTP request.

10. A system for automatically extracting news details from a large model using XPath, characterized in that, This system is used to implement the method for automatically extracting the XPath of the news detail page based on the large model as described in any one of claims 1-9. This system includes: A model evaluation module, used to evaluate multiple large models through multilingual recognition ability, long text reasoning ability, code execution ability and resource consumption, and determine the best large model base; A multilingual dataset construction module, used to collect list page HTML pages from multiple news websites, covering multiple languages, perform element positioning, data cleaning and cropping on the HTML pages, design a structured prompt template and construct a thought chain CoT dataset, and divide the dataset into a training set and a test set; An adaptive learning module, used to fine-tune the selected best large model base using the Lora technology, set the fine-tuning parameters, optimize the model using a smooth loss function until convergence, and introduce a dual reward mechanism for training, where the dual reward mechanism includes a format reward and a content reward; The format calibration module is used to correct the XPath expressions and JSON format of the model output using a format calibration tool to ensure that the output conforms to the expected structure. The XPath application and validation module is used to apply formatted XPath expressions to the HTML source code of the news list page, extract the details page URL, and validate the validity and accuracy of the URL.

Images