Hierarchical compound reward reinforcement learning-based large language model XPath generation method

Abstract

The application discloses a large language model XPath generation method based on layered composite reward reinforcement learning, and specifically comprises the following steps: step 1: obtaining the HTML source code and page element information of a target webpage and performing data cleaning to obtain structured data containing a DOM hierarchical order structure and element attribute values; performing data labeling on the structured data after data cleaning to obtain a labeled data set; step 2: selecting a basic model, using the labeled data set to supervise fine-tuning of the basic model, taking the basic model after the supervised fine-tuning as a strategy model; constructing a layered composite reward function to perform reinforcement learning fine-tuning, so that the output level of the strategy model is aligned with the input DOM level, and finally obtaining a final model after two-stage fine-tuning; step 3: generating a standard XPath string and outputting structured data matched with the input DOM level to show the layer-by-layer construction logic of the standard XPath string. The application can generate stable XPath and realize completely transparent and traceable generation process.

Description

Technical Field

[0001] This invention pertains to the field of computer technology, specifically relating to a method for generating XPath for large language models based on hierarchical composite reward reinforcement learning. Background Technology

[0002] In modern web development, automated testing, Robotic Process Automation (RPA), and web crawling, XPath serves as a crucial page element location technology. The stability, efficiency, and interpretability of its generated elements directly impact the performance of related systems. However, existing technologies suffer from several drawbacks: static attribute-based location methods rely on attributes such as ID and class, but face issues of poor stability, insufficient uniqueness, and high maintenance costs due to dynamic page attribute changes, attribute sharing, and page reconstruction; hierarchical location methods locate elements through DOM hierarchy relationships, but are vulnerable to changes in page structure and dynamically loaded content, making the location process fragile. The XPath generation methods suffer from several shortcomings. First, they are highly inefficient, and in deeply nested pages, the generated XPaths are lengthy, complex, and lack readability and maintainability. Second, text-based location methods rely on element text but suffer from poor dynamic adaptability and lack of uniqueness due to dynamically generated text and repetitive content. Third, manually written XPath methods require manual operation, resulting in inefficiency and reliance on user experience. Fourth, while machine learning-based automatic generation methods have made progress in recent years, they struggle to capture developers' attribute naming preferences, resulting in a black-box generation process lacking traceability and prone to hierarchical misalignment when processing deep DOM elements, failing to guarantee the integrity of the hierarchical structure. These shortcomings make XPath generation unsatisfactory in terms of stability, maintainability, and interpretability, necessitating innovative solutions. Summary of the Invention

[0003] The purpose of this invention is to provide a method for generating XPath for large language models based on hierarchical composite reward reinforcement learning. This invention can generate stable XPaths and achieve complete transparency and traceability in the generation process.

[0004] The technical solution of this invention: A method for generating XPath for large language models based on hierarchical composite reward reinforcement learning, specifically including the following steps:

[0005] Step 1: Obtain the HTML source code and page element information of the target webpage and perform data cleaning to obtain structured data containing DOM hierarchy and element attribute values; based on the existing large language model, combine manual annotation to annotate the cleaned structured data to obtain an annotated dataset;

[0006] Step 2: Select a base model and use the labeled dataset to perform supervised fine-tuning on the base model. Train the base model to select stable attribute values ​​from the element attribute values ​​of each DOM hierarchical structure. Use the base model after supervised fine-tuning as the policy model. Construct a hierarchical compound reward function for reinforcement learning fine-tuning. Use the GRPO algorithm to optimize the policy model to maximize the compound reward, so that the output level of the policy model is aligned with the input DOM level, and obtain the final model after two-stage fine-tuning.

[0007] Step 3: Based on the final model that has been fine-tuned in two stages, generate a standard XPath string and output structured data that matches the input DOM hierarchy to demonstrate the layer-by-layer construction logic of the standard XPath string.

[0008] In the aforementioned method for generating large language models using XPath based on hierarchical composite reward reinforcement learning, in step 1, the HTML source code and page element information of the target webpage are obtained using a web crawler; the data cleaning uses the BeautifulSoup library to remove ad tags and script content; the structured data format is a list containing key-value pairs of DOM hierarchy node attributes, and the list length represents the hierarchy depth of the elements.

[0009] In the aforementioned method for generating XPath for a large language model based on hierarchical compound reward reinforcement learning, the data annotation process involves using prompt word engineering to drive the large language model to filter stable and unstable attribute values. The labeled dataset includes the element attribute values ​​after data cleaning, stable attribute values ​​with business meaning annotated by the large language model, and unstable attribute values ​​that are not recommended for use.

[0010] In the aforementioned method for generating XPath for a large language model based on hierarchical composite reward reinforcement learning, in step 2, the base model is Qwen-3-1.7B, and the supervised fine-tuning converts the cleaned list of element attribute key-value pairs into a serialized input; then, the stable attribute values ​​labeled by the existing large language model are used as the target output of the base model, and the output data is a list of attribute values ​​with the same hierarchy as the input list.

[0011] In the aforementioned method for generating XPath for large language models based on hierarchical composite reward reinforcement learning, the training hyperparameters are set to a batch size of 32, a learning rate of 2e-5 with cosine annealing scheduling, a training period of 10 epochs, and a maximum sequence length of 512. The training process adopts an autoregressive generation mode to generate multi-level attribute value selection results, uses the cross-entropy loss function to optimize the prediction ability of the base model, calculates the prediction error of each level attribute value, and sets a dynamic masking mechanism to randomly mask some level attribute values ​​during training to enhance the robustness of the base model to incomplete inputs.

[0012] In the aforementioned method for generating large language models XPath based on hierarchical composite reward reinforcement learning, a reward model is constructed simultaneously during the training of the base model to evaluate the stability and business relevance of the attribute value list.

[0013] The hierarchical composite reward function is as follows:

[0014] ;

[0015] In the formula, This indicates the final reward. , and Represents the weight, where, It has the highest priority; This indicates a structural alignment bonus, generated by comparing the number of levels in the list of attribute values. With the number of input DOM path levels The matching degree is obtained; This represents the reward for attribute value selection preference, obtained by evaluating the list of attribute values ​​output by the policy model based on the reward model. This represents the conciseness penalty, calculated based on the length of the output sequence.

[0016] In the aforementioned XPath generation method for large language models based on hierarchical composite reward reinforcement learning, the optimization process includes the following steps:

[0017] Step 2.1: The policy model generates a complete list of attribute values ​​based on the current input;

[0018] Step 2.2: Calculate the final reward generated this time using a hierarchical composite reward function. ;in Obtained by calling the reward model;

[0019] Step 2.3: Calculate the dominance value of the action selection based on the attribute value at each level using generalized dominance estimation;

[0020] Step 2.4: The GRPO algorithm updates the network parameters of the policy model based on the calculated advantage value and regularization mechanism.

[0021] In the aforementioned method for generating XPath for large language models based on hierarchical composite reward reinforcement learning, step 3 generates a standard XPath string while outputting structured data that matches the input DOM hierarchy, which is used to demonstrate the hierarchical construction logic of the standard XPath string.

[0022] Compared with the prior art, the present invention has the following beneficial effects:

[0023] This invention learns from manually labeled expert data during a supervised fine-tuning phase, combined with reinforcement learning to solidify attribute value selection preferences. This enables the model to accurately capture developers' attribute value naming habits, prioritizing stable and business-meaningful attribute values. This effectively solves the problem of unstable positioning caused by attribute value changes and sharing on dynamic pages, reducing maintenance costs associated with page reconstruction. This invention introduces a hierarchical composite reward function with structural alignment rewards as the highest priority. By comparing the matching degree between the number of levels in the generated attribute value list and the number of levels in the input DOM path, it forces the policy model output to be strictly aligned with the input DOM levels. This solves the hierarchical misalignment problem that easily occurs when processing deep DOMs in existing technologies, ensuring the structural integrity and rigor of XPath generation. This invention, through the GRPO algorithm combined with a hierarchical composite reward function, effectively balances the optimization process of two types of learning signals: "hard structural rules" (hierarchical alignment) and "soft content preferences" (the rationality of attribute value selection). It ensures both the rigidity of structural constraints and the intelligence of attribute value selection, preventing policy collapse. This gives the policy model both structural rigor and content adaptability, making it suitable for complex dynamic page scenarios. This invention generates a standard XPath string and outputs structured data that matches the input DOM hierarchy, intuitively demonstrating the layer-by-layer construction logic of XPath. This allows users to clearly understand the model's attribute value selection decision process, greatly improving the system's transparency, reliability, and maintainability. Attached Figure Description

[0024] Figure 1 This is the method flow of the present invention;

[0025] Figure 2 This is a diagram showing the standard XPath string output of this invention;

[0026] Figure 3 The output diagram is a structured data set that matches the input DOM hierarchy. Detailed Implementation

[0027] The present invention will be further described below with reference to the accompanying drawings and embodiments, but this should not be construed as limiting the present invention.

[0028] Example: A method for generating XPath for large language models based on hierarchical composite reward reinforcement learning, such as... Figure 1 As shown, the specific steps include:

[0029] Step 1: Data Preparation and Model Initialization: Obtain the HTML source code and page element information of the target webpage and perform data cleaning to obtain structured data containing DOM hierarchical order structure and element attribute values; Based on the existing large language model, combine manual annotation to annotate the cleaned structured data to obtain an annotated dataset;

[0030] In this step, the HTML source code and page element information of the target webpage are obtained using web crawling tools. Alternatively, user behavior strategies such as using proxy IPs, setting request headers, and simulating page clicks or scrolling can be employed to acquire data. Data cleaning utilizes the BeautifulSoup library to remove useless information such as ad tags and script content, thereby improving the processing efficiency of the large language model. The structured data format is a list containing key-value pairs of DOM hierarchy node attributes, with the list length representing the element's hierarchy depth.

[0031] The data annotation process involves using prompt word engineering to drive a large language model to filter stable and unstable attribute values. The large language model used is GPT-4o, Claude 3.5, and Gemini 2.0. Although commercial large models possess powerful semantic understanding capabilities and can generate high-quality annotated data, generating XPaths for elements is a frequent requirement, and in practical applications, commercial large models do not meet timeliness requirements. Therefore, a batch of data that conforms to human preferences needs to be manually annotated. The annotated dataset includes cleaned element attribute values, stable attribute values ​​with business meaning annotated by the large language model, and unstable attribute values ​​that are not recommended for use.

[0032] In this embodiment, a product details page of a mainstream e-commerce platform is selected as the target webpage. The Scrapy crawler is used to crawl the page's HTML source code and the associated DOM hierarchy structure of the "Product Price" element (corresponding to the price text wrapped in a tag on the page). A proxy IP pool and request headers are configured during crawling to ensure data integrity. The BeautifulSoup4 library in Python is used to clean the crawled HTML source code: removing ad tags (filtering nodes containing ad identifiers and their child nodes); removing script content (deleting all tags and internal code); and formatting attribute values ​​(removing empty attribute values ​​and redundant spaces, retaining valid attribute values). After cleaning, structured data containing the DOM hierarchy structure and element attribute values ​​is generated. The format is a list of key-value pairs of hierarchy node attributes, with the list length corresponding to the element's hierarchy depth (in this scenario, the DOM hierarchy depth of the "Product Price" element is 3). The output of the large language model includes: stable attribute values: "data-page-type" (level 1), "prince-container" (level 2), "price" (level 3); unstable attribute values: "12345" (level 2, random suffix), "20250819"99.00"" (level 3, including dynamic date).

[0033] The final labeled dataset structure is shown in Table 1:

[0034] Table 1. Final Annotated Dataset Structure

[0035] Cleaned element attribute values Stable attribute values ​​(including business implications) Unstable property values ​​(not recommended) Level 1: div ["data-page-type"] "data-page-type" / Level 2: div ["price-container-12345"] "price-container" " 12345" Level 3: span ["price-20250819", "99.00"] "price" 20250819"99.00"

[0036] Step 2: Two-stage model fine-tuning: Select a base model and perform supervised fine-tuning on the base model using the labeled dataset. The training objective of the base model is to select stable attribute values ​​from the element attributes of each DOM hierarchical structure. The base model after supervised fine-tuning (SFT) is used as the policy model (SFT model). Construct a hierarchical composite reward function for reinforcement learning fine-tuning. Use the Generalized Regularized Policy Optimization (GRPO) algorithm to optimize the policy model. This algorithm avoids policy collapse through regularization mechanism, balances structural constraints and attribute selection intelligence, thereby maximizing composite rewards and aligning the output hierarchy of the policy model with the input DOM hierarchy, resulting in the final model after two-stage fine-tuning.

[0037] In this step, the core objective of the supervised fine-tuning phase is to use high-quality labeled data to teach the base model to understand and mimic the preferences and habits of human experts in selecting stable and meaningful HTML attribute values. The base model used is Qwen-3-1.7B. The supervised fine-tuning converts the cleaned element attributes into serialized input, where the input data is a concatenated string of hierarchical node attribute key-value pairs while preserving the DOM hierarchy order. Then, stable attribute values ​​labeled by the existing large language model are used as the target output of the base model, with the output data being a list consistent with the input data hierarchy. The training hyperparameters are set to a batch size of 32, a learning rate of 2e-5 using cosine annealing scheduling, a training period of 10 epochs, and a maximum sequence length of 512. The training process employs an autoregressive generation mode to generate multi-level attribute value selection results. The cross-entropy loss function is used to optimize the predictive ability of the base model, calculating the prediction error for each level of attribute value. A dynamic masking mechanism is also set up to include 10%-30% of non-critical level attribute values ​​(such as temporary attribute values ​​without business meaning) during training to enhance the robustness of the base model to incomplete input. After SFT fine-tuning, the basic model performed excellently in content selection, accurately identifying business abbreviations (such as ulgt), and successfully solved the problem of "difficulty in capturing attribute value naming preferences".

[0038] To address the hierarchical misalignment problem left over from the SFT stage that hinders the realization of core system functions, this invention introduces the Reinforcement Learning Fine-Tuning (RLHF) stage. This stage achieves precise control over the behavior of the policy model through two core steps: constructing reward signals and optimizing the execution policy.

[0039] During the training of the base model, a reward model is simultaneously constructed to evaluate the stability and business relevance of the attribute value list.

[0040] The hierarchical composite reward function designed in this invention This is the final signal driving the optimization of the policy model. The reward function itself has an inherent, priority-based logical hierarchy, designed to guide the policy model to follow the optimization path of "structure first, content second," combining objective structural rules with subjective content preferences.

[0041] The hierarchical composite reward function is as follows:

[0042] ;

[0043] In the formula, This indicates the final reward. , and Represents the weight, where, It has the highest priority; This indicates a structural alignment bonus, generated by comparing the number of levels in the list of attribute values. With the number of input DOM path levels The matching degree is obtained. If they are equal, a high positive reward is given; otherwise, a significant negative penalty is given. This represents the reward for attribute value selection preference, obtained by evaluating the list of attribute values ​​output by the policy model based on the reward model. This represents the conciseness penalty, calculated based on the length of the output sequence.

[0044] The optimization process includes the following steps:

[0045] Step 2.1 Experience Collection: The strategy model generates a complete list of attribute values ​​based on the current input;

[0046] Step 2.2 Reward Calculation: Calculate the final reward generated in this step using a hierarchical composite reward function. ;in Obtained by calling the reward model;

[0047] Step 2.3 Advantage Estimation: Use Generalized Advantage Estimation (GAE) to calculate the advantage value of the action selection from the attribute value list at each level;

[0048] Step 2.4 Policy Update: The GRPO algorithm updates the network parameters of the policy model based on the calculated advantage value and regularization mechanism. The stability advantage of the GRPO algorithm effectively prevents the model from pursuing high advantages. When rewarding, forget the attribute value selection ability learned in the SFT phase.

[0049] Through the above steps, the strategy model, guided by the GRPO algorithm, can learn to strictly adhere to hierarchical matching rules (to obtain...). At the same time, select attribute values ​​that are preferred by humans as much as possible (to obtain...) ).

[0050] In this embodiment, GPT-4o is used as the annotation-driven model, and the prompt word engineering instruction is designed as follows: "The user provides a JSON representation of a set of element DOM information from the target node to the root node in the DOM. For example, \nSelect all long-term stable attribute values.\n\nAttribute value requirements:\n1. Have clear semantics, or have clear business meaning or interaction intent.\n2. Not a dynamic or random value of random character combinations.\n\nNote:\n1. Consider the detailed reasons for retaining or discarding each attribute value of each attribute at each level of the node, and ensure that all attribute values ​​that meet the conditions are retained.\n2. Ensure that the output results correspond in level, and do not omit or add levels.\n3. Support the selection of substrings that meet the requirements in the attribute values.\n4. If no matching attribute is found for a node, only the node name is returned.\n5. Only output the Python result array, and do not include other information."

[0051] In this embodiment, Qwen-3-1.7B is selected as the base model. The structured data generated in step 1 is converted into serialized input: Input data (a concatenated string that retains the DOM hierarchy order). Target output (a list of hierarchical tags consistent with the XPath generation logic): [Level 1: Select attribute 'data-page-type'] → [Level 2: Select attribute 'price-container'] → [Level 3: Select attribute 'price']. Hyperparameters are set as follows: training batch size is 32; learning rate is 2e-5 (using cosine annealing scheduling); training epoch is 10; early stopping strategy (stop if the validation set loss does not decrease for 3 consecutive epochs); maximum sequence length is 512; training mode uses autoregressive generation, and the cross-entropy loss function is used to calculate the prediction error of each attribute (e.g., if level 2 predicts "price-container-12345", the error will increase); dynamic masking mechanism: randomly mask 15% of non-critical attributes (e.g., "style" in level 2, "data-page-type" in level 1) to enhance the model's robustness to incomplete input. After supervised fine-tuning, the model can stably output a hierarchical list of "prioritizing business-related static attributes," achieving an accuracy of 92% without masking. The reward model, trained using a labeled dataset, has the core function of: taking the attribute value list generated by the policy model as input, and outputting a "stability score (0-5 points)" and a "business relevance score (0-3 points)." The weighted sum of these two scores is the result. (The weights, verified through manual annotation and statistical analysis, are 0.7 and 0.3 respectively). For example:

[0052] Attribute value list:

[0053] "[['data-page-type'],['price-container'],['price']]": Stability score 5, business relevance score 3. =4.4;

[0054] Attribute value list:

[0055] "[['data-page-type'],['price-container-12345'],['99.00']]": Stability score 1, Business relevance score 1. =1.0.

[0056] Then set the weights. =0.6 (Structure alignment bonus, highest priority) =0.3 (Attribute Selection Preference Reward) =0.1 (simplicity penalty), with a specific calculation example as follows:

[0057] Structure alignment reward ( ): Input DOM hierarchy number =3, number of levels in the attribute value list generated by the strategy model =3, matching degree 100%, given =+5; if =2, then give =-10;

[0058] Attribute selection preference reward ( ): Take 4.4 of the reward model output;

[0059] Simplicity penalty ( ): The output attribute value list has 68 characters (not exceeding the threshold of 80), so give R =-0.5 (0.5 deducted for every 10 characters exceeding the limit);

[0060] The final reward is calculated. =4.27.

[0061] In this embodiment, the optimization process of the GRPO algorithm is as follows:

[0062] Step 2.1 (Experience Collection): Input the above e-commerce DOM structured data into the strategy model and generate a list of attribute values ​​"[['data-page-type'],[ 'price-container'],[ 'price']]";

[0063] Step 2.2 (Reward Calculation): Obtain the reward using the formula above. =4.27;

[0064] Step 2.3 (Dominance Estimation): Use Generalized Dominance Estimation (GAE) to calculate the dominance value for each attribute value selection. For example, the dominance value for selecting "price-container" in level 2 is 2.8 (higher than the 0.5 for selecting "price-container-12345").

[0065] Step 2.4 (Policy Update): The GRPO algorithm adjusts the network parameters of the policy model based on the dominance value and regularization mechanism to ensure that the updated policy model still retains the ability to "prefer static attribute values" learned in the SFT stage, while strengthening the "hierarchy alignment" constraint (such as increased loss when the number of hierarchies deviates).

[0066] Step 3: Model Application and Output: Based on the final model that has been fine-tuned in two stages, generate a standard XPath string and output structured data that matches the input DOM hierarchy to demonstrate the layer-by-layer construction logic of the standard XPath string.

[0067] In this step, the final model is encapsulated into an XPath-generated intelligent agent module (Agent). This Agent receives DOM structured data and performs dual outputs:

[0068] Output 1: A highly stable standard XPath string generated based on the selected attribute value, such as... Figure 2 As shown;

[0069] Output 2: A structured JSON or list of data that perfectly matches the input DOM hierarchy. The structured output data can be in JSON format and clearly shows the specific attribute values ​​used at each level when generating the XPath, such as... Figure 3 As shown.

[0070] The system presents "Output 1" and "Output 2" to the user in a visual way, allowing the user to understand the construction logic of XPath at a glance and to analyze or adjust the selection strategy of specific levels when needed.

[0071] In this embodiment, the final model, after two stages of fine-tuning, receives the DOM structured data of the "product price" element in the new webpage and generates a standard XPath string. This XPath can reliably locate the product price element, and even if the page is restructured (such as dynamic class updates), it remains valid due to its static attribute values. Compared to traditional XPath based on dynamic classes, the XPath generated by this method maintains a 100% success rate in locating the element after daily page updates (the traditional method drops to 35%); when processing 100 deep DOM elements (level ≥ 5) of e-commerce elements, the level misalignment rate of this method is 0 (the misalignment rate of traditional machine learning methods is 22%); through structured data output, the user's understanding cost of XPath construction logic is reduced by 70%, eliminating the need for manual reverse derivation of the attribute value selection criteria.

[0072] In summary, this invention can generate stable XPath and achieve a completely transparent and traceable generation process.

Claims

1. A method for generating XPath for large language models based on hierarchical composite reward reinforcement learning, characterized in that, Specifically, the following steps are included: Step 1: Obtain the HTML source code and page element information of the target webpage and perform data cleaning to obtain structured data containing DOM hierarchy and element attribute values; Based on a large language model combined with manual annotation, the cleaned structured data is annotated to obtain an annotated dataset. Step 2: Select a base model and use the labeled dataset to perform supervised fine-tuning on the base model. Train the base model to select stable attribute values ​​from the element attribute values ​​of each DOM hierarchical structure. Use the base model after supervised fine-tuning as the policy model. Construct a hierarchical compound reward function for reinforcement learning fine-tuning. Use the GRPO algorithm to optimize the policy model to maximize the compound reward, so that the output level of the policy model is aligned with the input DOM level, and obtain the final model after two-stage fine-tuning. Step 3: Based on the final model that has been fine-tuned in two stages, generate a standard XPath string and output structured data that matches the input DOM hierarchy to demonstrate the layer-by-layer construction logic of the standard XPath string; In step 2, a reward model is built simultaneously during the training of the base model to evaluate the stability and business relevance of the attribute value list. The hierarchical composite reward function is as follows: ； In the formula, This indicates the final reward. , and Represents the weight, where, It has the highest priority; This indicates a structural alignment bonus, generated by comparing the number of levels in the list of attribute values. With the number of input DOM path levels The matching degree is obtained; This represents the reward for attribute value selection preference, obtained by evaluating the list of attribute values ​​output by the policy model based on the reward model. This represents the conciseness penalty, calculated based on the length of the output sequence.

2. The method for generating XPath for large language models based on hierarchical composite reward reinforcement learning according to claim 1, characterized in that: In step 1, the HTML source code and page element information of the target webpage are obtained using a web crawler; the data cleaning uses the BeautifulSoup library to remove ad tags and script content; the structured data format is a list containing key-value pairs of DOM hierarchy node attributes, and the list length represents the hierarchy depth of the elements.

3. The method for generating XPath for large language models based on hierarchical composite reward reinforcement learning according to claim 1, characterized in that: The data annotation process involves using prompt word engineering to drive the large language model to filter stable and unstable attribute values. The annotated dataset includes the element attribute values ​​after data cleaning, stable attribute values ​​with business meaning annotated by the large language model, and unstable attribute values ​​that are not recommended for use.

4. The method for generating XPath for large language models based on hierarchical composite reward reinforcement learning according to claim 1, characterized in that: In step 2, the supervised fine-tuning converts the cleaned list of element attribute key-value pairs into serialized input; then, the stable attribute values ​​labeled by the large language model are used as the target output of the basic model, and the output data is a list of attribute values ​​with the same hierarchy as the input list.

5. The method for generating XPath for large language models based on hierarchical composite reward reinforcement learning according to claim 1, characterized in that: The training hyperparameters are set to a batch size of 32, a learning rate of 2e-5 with cosine annealing scheduling, a training period of 10 epochs, and a maximum sequence length of 512. The training process adopts an autoregressive generation mode to generate multi-level attribute value selection results, uses the cross-entropy loss function to optimize the prediction ability of the base model, calculates the prediction error of each level attribute value, and sets a dynamic masking mechanism to randomly mask some level attribute values ​​during training to enhance the robustness of the base model to incomplete inputs.

6. The method for generating XPath for large language models based on hierarchical composite reward reinforcement learning according to claim 1, characterized in that: The optimization process includes the following steps: Step 2.1: The policy model generates a complete list of attribute values ​​based on the current input; Step 2.2: Calculate the final reward generated this time using a hierarchical composite reward function. ;in Obtained by calling the reward model; Step 2.3: Calculate the dominance value of the action selection from the attribute value list at each level using generalized dominance estimation; Step 2.4: The GRPO algorithm updates the network parameters of the policy model based on the calculated advantage value and regularization mechanism.

7. The method for generating XPath for large language models based on hierarchical composite reward reinforcement learning according to claim 1, characterized in that: In step 3, while generating the standard XPath string, a structured data matching the input DOM hierarchy is output to demonstrate the hierarchical construction logic of the standard XPath string.

Images