Effect evaluation method for generating user interface element selector based on large model

Abstract

The invention discloses an effect evaluation method for generating a user interface element selector based on a large model, and the method specifically comprises the following steps: S1, marking a target element by a user, capturing page information, obtaining a complete DOM (Document Object Model) structure, and extracting element features from the obtained DOM structure; s2, submitting the extracted element features to a large language model, wherein the large language model generates various CSS selectors based on the obtained element features; s3, multi-dimensional evaluation is carried out on the generated selector, comprehensive quality score calculation is carried out on the selector through multi-dimensional evaluation, and the multi-dimensional evaluation comprises accuracy scoring, stability scoring and performance scoring; and S4, performing an automatic retry generation mechanism based on an evaluation result to obtain a selector with the highest quality score. According to the method, diversified selector candidates based on different characteristics are generated through multi-model selector generation and by using LLM analysis element characteristics.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology, specifically relating to a method for evaluating the effect of generating user interface element selectors based on large models. Background Technology

[0002] In applications such as RPA (Robotic Process Automation) and Web automation, accurately, stably, and efficiently locating target elements on web pages is crucial for task success. Traditional CSS selector generation relies primarily on static rules or manual configuration. However, in modern web applications with frequently changing page structures and diverse attributes, this often leads to selector failures, inaccurate positioning, and low efficiency, severely impacting the robustness and maintenance costs of automated processes. In recent years, breakthroughs in Natural Language Modeling (LLM) in natural language understanding and structured data processing have provided a new technological path for intelligent selector generation. By introducing LLM's ability to understand DOM structure and element semantic features, combined with automatic evaluation and optimization mechanisms, automatic generation, scoring, stability testing, and iterative optimization of selectors can be achieved, significantly improving their accuracy and robustness. This selector generation and evaluation system, integrating AI capabilities, offers significant advantages in complex page environments and is a crucial foundational technology for building intelligent automated systems. Summary of the Invention

[0003] The purpose of this invention is to overcome the shortcomings of the prior art and provide a method for evaluating the effect of generating user interface element selectors based on large models. The method of this invention generates multi-model selectors and uses LLM to analyze element features to generate diverse selector candidates based on different features.

[0004] To achieve the above-mentioned objectives, the technical solution provided by this invention patent is as follows:

[0005] A method for evaluating the effectiveness of generating user interface element selectors based on large models, the method specifically includes the following steps:

[0006] S1, the user marks the target element, captures page information, obtains the complete DOM structure, and extracts element features from the obtained DOM structure;

[0007] S2, the extracted element features are submitted to the large language model, which generates a variety of CSS selectors based on the obtained element features;

[0008] S3 performs multi-dimensional evaluation on the generated selector and calculates a comprehensive quality score for the selector through multi-dimensional evaluation, which includes accuracy score, stability score and performance score.

[0009] S4 is an automatic retry generation mechanism based on the evaluation results to obtain the selector with the highest quality score.

[0010] Furthermore, the extracted element features include label type, attributes, hierarchical relationships, and the relative position and semantic relationships between the target element and surrounding elements.

[0011] Furthermore, the accuracy score evaluates whether the selector can uniquely and accurately locate the target element within the current page and foreseeable page changes. Specifically, the accuracy score is as follows:

[0012] The uniqueness test executes a selector on the target page and checks the number of returned DOM elements. A uniqueness score is obtained based on the number of returned DOM elements. When 1 element is returned, the uniqueness score is 100 points; when 2-3 elements are returned, the uniqueness score is 60 points; when 4-10 elements are returned, the uniqueness score is 30 points; and when more than 10 elements are returned, the uniqueness score is 0 points.

[0013] Element matching verification compares the feature signatures of the elements returned by the selector with those of the expected target element. The matching feature count determination method is that if the similarity between the element returned by the selector and the expected target element is ≥90%, it is considered a matching feature. The feature signature similarity algorithm specifically constructs the feature vector of the target element, including the label name, ID, class name, and text content. Weights are assigned to each feature, and the feature similarity between the element returned by the selector and the target element is calculated: similarity = Σ(matching feature × weight) / Σ(all feature weights). The matching score is obtained based on the number of matching features = (number of matching features / total number of features) × 100.

[0014] The accuracy score is obtained based on the uniqueness score and the matching score: Accuracy score = (uniqueness score × 0.6) + (matching score × 0.4).

[0015] Furthermore, the stability score assesses the selector's durability and adaptability under changes in page structure and attributes; the stability score specifically includes:

[0016] Stability score = (structural stability × 0.5) + (attribute stability × 0.5);

[0017] Wherein, structural stability = (pass rate of slight change × 0.5) + (pass rate of moderate change × 0.3) + (pass rate of drastic change × 0.2);

[0018] Slight changes to the DOM structure involve adding or deleting non-critical nodes; moderate changes involve adjusting hierarchical relationships; drastic changes involve altering the parent node structure.

[0019] Attribute stability = (Class name change pass rate × 0.4) + (ID change pass rate × 0.4) + (Content change pass rate × 0.2);

[0020] Class name change test involves adding, removing, or modifying class names; ID change test involves modifying non-critical IDs; content change test involves changes to text content.

[0021] Furthermore, the performance score is used to evaluate the efficiency and complexity of the selector execution;

[0022] Performance score = (Speed ​​score × 0.7) + (Complexity score × 0.3)

[0023] Execution time measurement: The selector execution time is recorded using a high-precision timer. The test is repeated on pages with different DOM sizes. After excluding outliers, the average value is taken. A speed score is calculated based on the obtained execution time.

[0024] Speed ​​score = 100 - min(100, (actual execution time / baseline execution time) × 50)

[0025] Complexity analysis involves analyzing the structural complexity of the selector, including nesting levels and the number of conditions, estimating the computational complexity of selector execution, and assessing the blocking impact of the selector on the browser rendering thread. A complexity score is then calculated: Complexity Score = 100 - (Selector Complexity Index × 10).

[0026] Selector complexity index = base complexity + Σ (complexity contribution of each part), where base complexity = number of selector segments × 1.0, ID selector contribution = number of occurrences × 0.1, class selector contribution = number of occurrences × 0.3, attribute selector contribution = number of occurrences × 0.5, pseudo-class selector contribution = number of occurrences × 0.7, wildcard contribution = number of occurrences × 1.0, child selector contribution = number of occurrences × 0.5, and descendant selector contribution = number of occurrences × 0.8.

[0027] Furthermore, the calculation of the overall quality score is as follows:

[0028] Overall Quality Score = (Accuracy Score × W_Accuracy) + (Stability Score × W_Stability) + (Performance Score × W_Performance)

[0029] Where W_accuracy + W_stability + W_performance = 1, and the weight configuration is W_accuracy = 0.5, W_stability = 0.4, and W_performance = 0.1.

[0030] Furthermore, the automatic retry generation mechanism based on the evaluation results is as follows: when the overall quality score of all generated selectors is less than 60, repeat steps S1-S3 to generate selectors; until the quality score requirement or the maximum number of iterations is reached, and finally output the selector with the highest score.

[0031] Based on the above technical solutions, the method for evaluating the effect of generating user interface element selectors based on large models, as proposed in this invention, has achieved the following technical advantages through practical application:

[0032] 1. The present invention provides an effect evaluation method for generating user interface element selectors based on large models. By generating multi-model selectors and using LLM to analyze element features, a variety of selector candidates based on different features are generated.

[0033] 2. The present invention provides an evaluation method for the effect of a user interface element selector based on a large model. Through comprehensive evaluation dimensions, the accuracy evaluation ensures that the selector can uniquely and accurately locate the target element, the stability evaluation tests the robustness of the selector by simulating various page changes, and the performance evaluation ensures that the selector executes efficiently and does not affect page performance.

[0034] 3. The present invention provides an effect evaluation method for a user interface element selector based on a large model. It uses a weight system to reflect the importance of different factors through a scientific scoring mechanism. The structured scoring process ensures the objectivity and repeatability of the evaluation. The adaptive weight mechanism can adjust the scoring focus according to different scenarios. Attached Figure Description

[0035] Figure 1 This is a flowchart of the effect evaluation process in the effect evaluation method of generating user interface element selectors based on large models according to the present invention. Detailed Implementation

[0036] To make the objectives, technical solutions, and advantages of this invention clearer, the invention is described below with reference to specific examples shown in the accompanying drawings. However, it should be understood that these descriptions are merely exemplary and not intended to limit the scope of the invention. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concept of the invention.

[0037] Example 1

[0038] like Figure 1 As shown, a method for evaluating the effectiveness of a user interface element selector generated based on a large model is presented. This method specifically includes the following steps:

[0039] S1, the user marks the target element, captures page information, obtains the complete DOM structure, and extracts element features from the obtained DOM structure;

[0040] S2, the extracted element features are submitted to the large language model, which generates a variety of CSS selectors based on the obtained element features;

[0041] S3 performs multi-dimensional evaluation on the generated selector and calculates a comprehensive quality score for the selector through multi-dimensional evaluation, which includes accuracy score, stability score and performance score.

[0042] S4 is an automatic retry generation mechanism based on the evaluation results to obtain the selector with the highest quality score.

[0043] The extracted element features include label type, attributes, hierarchical relationships, and the relative position and semantic relationships between the target element and surrounding elements.

[0044] The accuracy score evaluates whether the selector can uniquely and accurately locate the target element within the current page and foreseeable page changes. Specifically, the accuracy score is as follows:

[0045] The uniqueness test executes a selector on the target page and checks the number of returned DOM elements. A uniqueness score is obtained based on the number of returned DOM elements. When 1 element is returned, the uniqueness score is 100 points; when 2-3 elements are returned, the uniqueness score is 60 points; when 4-10 elements are returned, the uniqueness score is 30 points; and when more than 10 elements are returned, the uniqueness score is 0 points.

[0046] Element matching verification compares the feature signatures of the elements returned by the selector with those of the expected target element. The matching feature count determination method is that if the similarity between the element returned by the selector and the expected target element is ≥90%, it is considered a matching feature. The feature signature similarity algorithm specifically constructs the feature vector of the target element, including the label name, ID, class name, and text content. Weights are assigned to each feature, and the feature similarity between the element returned by the selector and the target element is calculated: similarity = Σ(matching feature × weight) / Σ(all feature weights). The matching score is obtained based on the number of matching features = (number of matching features / total number of features) × 100.

[0047] The accuracy score is obtained based on the uniqueness score and the matching score: Accuracy score = (uniqueness score × 0.6) + (matching score × 0.4).

[0048] The stability score assesses the selector's durability and adaptability under changes in page structure and attributes; the stability score is specifically as follows:

[0049] Stability score = (structural stability × 0.5) + (attribute stability × 0.5);

[0050] Wherein, structural stability = (pass rate of slight change × 0.5) + (pass rate of moderate change × 0.3) + (pass rate of drastic change × 0.2);

[0051] Slight changes to the DOM structure involve adding or deleting non-critical nodes; moderate changes involve adjusting hierarchical relationships; drastic changes involve altering the parent node structure.

[0052] Attribute stability = (Class name change pass rate × 0.4) + (ID change pass rate × 0.4) + (Content change pass rate × 0.2);

[0053] Page change simulation method: Use the DOM manipulation API to create a page variant tree, apply a structural transformation matrix to randomly generate variant versions, the transformation intensity follows a power law distribution, with high probability of small changes and low probability of large changes.

[0054] Class name change test involves adding, removing, or modifying class names; ID change test involves modifying non-critical IDs; content change test involves changes to text content.

[0055] The performance score is used to evaluate the efficiency and complexity of the selector execution;

[0056] Performance score = (Speed ​​score × 0.7) + (Complexity score × 0.3)

[0057] Execution time measurement: The selector execution time is recorded using a high-precision timer. The test is repeated on pages with different DOM sizes. After excluding outliers, the average value is taken. A speed score is calculated based on the obtained execution time.

[0058] Speed ​​score = 100 - min(100, (actual execution time / baseline execution time) × 50)

[0059] Complexity analysis involves analyzing the structural complexity of the selector, including nesting levels and the number of conditions, estimating the computational complexity of selector execution, and assessing the blocking impact of the selector on the browser rendering thread. A complexity score is then calculated: Complexity Score = 100 - (Selector Complexity Index × 10).

[0060] Selector complexity index = base complexity + Σ (complexity contribution of each part), where base complexity = number of selector segments × 1.0, ID selector contribution = number of occurrences × 0.1, class selector contribution = number of occurrences × 0.3, attribute selector contribution = number of occurrences × 0.5, pseudo-class selector contribution = number of occurrences × 0.7, wildcard contribution = number of occurrences × 1.0, child selector contribution = number of occurrences × 0.5, and descendant selector contribution = number of occurrences × 0.8.

[0061] The calculation of the overall quality score is as follows:

[0062] Overall Quality Score = (Accuracy Score × W_Accuracy) + (Stability Score × W_Stability) + (Performance Score × W_Performance)

[0063] Where W_accuracy + W_stability + W_performance = 1, and the weight configuration is W_accuracy = 0.5, W_stability = 0.4, and W_performance = 0.1.

[0064] Scenario-adaptive weighting mechanism: 1. Websites with high-frequency changes: Increase stability and robustness weights; 2. Performance-sensitive scenarios: Increase performance weights; 3. Precise operation scenarios: Increase precision weights.

[0065] The automatic retry generation mechanism based on the evaluation results is as follows: when the overall quality score of all generated selectors is less than 60, repeat steps S1-S3 to generate selectors; until the quality score requirement or the maximum number of iterations is reached, and finally output the selector with the highest score.

[0066] The beneficial effects of this invention include: the ability to simulate page changes: by creating page variants through the DOM manipulation API, the system can simulate various common DOM structures and attribute changes, and the test results directly reflect the reliability of the selector in the real environment.

[0067] Data-driven selector optimization: Make selections based on quantitative scores rather than subjective judgments, provide alternative solutions to meet the needs of different scenarios, and verify in practice to ensure that theoretical scores are consistent with actual performance.

[0068] Example 2

[0069] Case Study: E-commerce Website Product Detail Page Element Locator

[0070] Scene Description

[0071] Suppose we need to create a stable selector for an automated testing tool to locate the "Add to Cart" button on an e-commerce website. This button may be positioned similarly on different product pages, but the surrounding structure, style, and attributes may vary.

[0072] Implementation process

[0073] 1. User tagging and feature capture

[0074] Users open a product details page on an e-commerce website in their browser and use our tool to mark the "Add to Cart" button.

[0075] Captured DOM structure fragments:

[0076]

[0077] Extracted elemental features:

[0078]

[0079]

[0080] 2. Selector generation

[0081] The system submits the feature information to the LLM to generate multiple selector schemes:

[0082] Selector candidate 1 (based on ID):

[0083] #add-to-cart-btn-10234

[0084] Selector candidate 2 (based on class name):

[0085] .btn.primary-action.add-to-cart

[0086] Selector candidate 3 (based on text content):

[0087] button.btn:contains("Add to cart")

[0088] Selector candidate 4 (based on attributes and adjacency):

[0089] button[data-product-id].btn.add-to-cart

[0090] Selector candidate 5 (based on relative path and feature combination):

[0091] div.product-actions div.action-buttons>button.add-to-cart

[0092] 3. Selector Evaluation

[0093] The system performs multi-dimensional evaluation on each selector:

[0094] A. Accuracy Assessment

[0095] Execute each selector on the current page and test its accuracy:

[0096] The system performs multi-dimensional evaluation of each selector:

[0097] A. Accuracy Assessment

[0098]

[0099] B. Stability Assessment

[0100] The system generates page variants for testing:

[0101] Structural change simulation:

[0102] Minor changes: such as adding other buttons or altering non-critical nodes.

[0103] • Moderate changes: such as adjusting the hierarchy of parts of the DOM structure

[0104] Drastic changes: such as altering the parent node structure

[0105] Attribute change simulation:

[0106] • Class name changes: Adding / removing style classes

[0107] • ID Change: Modify the numeric portion of the ID.

[0108] • Content change: The button text changed from "Add to Cart" to "Add to Cart"

[0109] Test results:

[0110]

[0111] C. Performance Evaluation

[0112] Execution time measurement and complexity analysis:

[0113]

[0114] D. Calculation of overall score

[0115] Application weights: Accuracy (0.5), Stability (0.4), Performance (0.1)

[0116]

[0117]

[0118] 4. Results and Optimization Selection

[0119] Based on the comprehensive score, the system determined that button[data-product-id].btn.add-to-cart was the optimal selector, with a comprehensive score of 94.42.

[0120] The system also provides the following alternative solutions:

[0121] 1..btn.primary-action.add-to-cart (92.27 points)

[0122] 2.div.product-actions div.action-buttons>button.add-to-cart (87.80 points)

[0123] 5. Actual verification

[0124] Validate the optimal selector on 10 different product pages:

[0125] Product page Was the location successfully located? Remark Product 1 √ Exact match Product 2 √ Exact match Product 3 √ Exact match Product 4 √ Exact match Product 5 √ Exact match Product 6 √ Exact match Product 7 √ Exact match Product 8 √ Exact match Item 9 √ Accurate match, but page loads slowly. Item 10 √ Exact match

[0126] The optimal selector demonstrated 100% accuracy and stability in actual testing.

[0127] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them; although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications can still be made to the specific implementation of the invention or equivalent substitutions can be made to some technical features without departing from the spirit of the technical solutions of the present invention, and all such modifications and substitutions should be covered within the scope of the technical solutions claimed in the present invention.

Claims

1. A method for evaluating the effectiveness of user interface element selectors generated based on large models, characterized in that, The method specifically includes the following steps: S1, the user marks the target element, captures page information, obtains the complete DOM structure, and extracts element features from the obtained DOM structure; S2, the extracted element features are submitted to the large language model, which generates a variety of CSS selectors based on the obtained element features; S3 performs multi-dimensional evaluation on the generated selector and calculates a comprehensive quality score for the selector through multi-dimensional evaluation, which includes accuracy score, stability score and performance score. S4 is an automatic retry generation mechanism based on the evaluation results to obtain the selector with the highest quality score.

2. The method for evaluating the effect of a user interface element selector based on a large model, as described in claim 1, is characterized in that... The extracted element features include label type, attributes, hierarchical relationships, and the relative position and semantic relationships between the target element and surrounding elements.

3. The method for evaluating the effect of a user interface element selector based on a large model, as described in claim 1, is characterized in that... The accuracy score evaluates whether the selector can uniquely and accurately locate the target element within the current page and foreseeable page changes. Specifically, the accuracy score is as follows: The uniqueness test executes a selector on the target page and checks the number of returned DOM elements. A uniqueness score is obtained based on the number of returned DOM elements. When 1 element is returned, the uniqueness score is 100 points; when 2-3 elements are returned, the uniqueness score is 60 points; when 4-10 elements are returned, the uniqueness score is 30 points; and when more than 10 elements are returned, the uniqueness score is 0 points. Element matching verification compares whether the feature signature of the element returned by the selector matches that of the expected target element. The matching feature determination method is that if the similarity between the element returned by the selector and the expected target element is ≥90%, it is considered a matching feature; The feature signature similarity algorithm specifically constructs a feature vector of the target element, including the label name, ID, class name, and text content; assigns weights to each feature, and calculates the feature similarity between the selector-returned element and the target element: similarity = Σ(matching features × weights) / Σ(all feature weights), and obtains the matching score based on the number of matching features = (number of matching features / total number of features) × 100; The accuracy score is obtained based on the uniqueness score and the matching score: Accuracy score = (uniqueness score × 0.6) + (matching score × 0.4).

4. The method for evaluating the effect of a user interface element selector based on a large model, as described in claim 1, is characterized in that... The stability score assesses the selector's durability and adaptability under changes in page structure and attributes; the stability score is specifically as follows: Stability score = (structural stability × 0.5) + (attribute stability × 0.5); Wherein, structural stability = (pass rate of slight change × 0.5) + (pass rate of moderate change × 0.3) + (pass rate of drastic change × 0.2); Slight changes to the DOM structure involve adding or deleting non-critical nodes; moderate changes involve adjusting hierarchical relationships; drastic changes involve altering the parent node structure. Attribute stability = (Class name change pass rate × 0.4) + (ID change pass rate × 0.4) + (Content change pass rate × 0.2); The class name change test involves adding, removing, or modifying the class name; the ID change test involves modifying a non-critical ID; and the content change test involves changes to the text content.

5. The method for evaluating the effect of a user interface element selector based on a large model, as described in claim 1, is characterized in that... The performance score is used to evaluate the efficiency and complexity of the selector execution; Performance score = (Speed ​​score × 0.7) + (Complexity score × 0.3) Execution time measurement: The selector execution time is recorded using a high-precision timer. The test is repeated on pages with different DOM sizes. After excluding outliers, the average value is taken. A speed score is calculated based on the obtained execution time. Speed ​​score = 100 - min(100, (actual execution time / baseline execution time) × 50). Complexity analysis analyzes the structural complexity of the selector, including nesting level and number of conditions, estimates the computational complexity of selector execution, and evaluates the blocking impact of the selector on the browser rendering thread to obtain the complexity score: Complexity score = 100 - (selector complexity exponent × 10). Selector complexity index = base complexity + Σ (complexity contribution of each part), where base complexity = number of selector segments × 1.0, ID selector contribution = number of occurrences × 0.1, class selector contribution = number of occurrences × 0.3, attribute selector contribution = number of occurrences × 0.5, pseudo-class selector contribution = number of occurrences × 0.7, wildcard contribution = number of occurrences × 1.0, child selector contribution = number of occurrences × 0.5, and descendant selector contribution = number of occurrences × 0.

8.

6. The method for evaluating the effect of a user interface element selector based on a large model, as described in claim 1, is characterized in that... The calculation of the overall quality score is as follows: Overall Quality Score = (Accuracy Score × W_Accuracy) + (Stability Score × W_Stability) + (Performance Score × W_Performance) Where W_accuracy + W_stability + W_performance = 1, and the weight configuration is W_accuracy = 0.5, W_stability = 0.4, and W_performance = 0.

1.

7. The method for evaluating the effect of a user interface element selector based on a large model, as described in claim 1, is characterized in that... The automatic retry generation mechanism based on the evaluation results is as follows: when the overall quality score of all generated selectors is less than 60, repeat steps S1-S3 to generate selectors; until the quality score requirement or the maximum number of iterations is reached, and finally output the selector with the highest score.

Images