An intelligent element positioning method and system based on AI and dynamic feature library

Abstract

The application discloses an intelligent element positioning method and system based on AI and a dynamic feature library, which comprises: abnormality detection and abnormal data capture, element positioning operation is performed through an automatic script, and a page DOM tree structure snapshot is recorded when positioning fails; multi-modal feature extraction, DOM tree analysis, extraction of XPath / CSS path and semantic attributes of a target element; dynamic feature library retrieval and matching, calculation of DOM structure similarity, and use of high similarity rules as a correction basis; AI intelligent rule generation, generation of candidate positioning rules through a classification model; script correction and real-time feedback, replacement of invalid positioning statements and updating of the dynamic feature library; and generation of a structured report, recording of abnormal data. The application realizes intelligent correction of element positioning in an automatic test process through the combination of AI and a dynamic feature library, and improves the stability and efficiency of automatic testing.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of automation testing, and in particular to an intelligent element positioning method and system based on AI and dynamic feature library. BACKGROUND

[0002] With the rapid development of Internet technology, the complexity of Web applications and mobile applications is continuously increasing, and automation testing plays an increasingly important role in the software development life cycle. In the process of automation testing, element positioning is a fundamental and critical link, and its stability and accuracy directly affect the reliability and maintenance cost of test scripts.

[0003] Currently, the main element positioning methods mainly include static positioning methods based on XPath, CSS selectors, ID, Name, etc. However, with the development of front-end technology, especially the widespread use of dynamic rendering, responsive design and single-page applications (SPA), traditional static positioning methods face many challenges.

[0004] The existing element positioning methods have the following technical problems: 1. Strong dependence on static rules: Most existing element positioning methods rely on static rules, and when element attributes or page structure change, scripts need to be manually modified, with high maintenance costs. Especially in agile development environments, frequent interface changes require continuous adjustment of automation test scripts, seriously affecting test efficiency.

[0005] 2. Poor adaptability to dynamic content: Existing technologies are difficult to effectively handle dynamically generated elements or complex UI interaction scenarios. In single-page applications (SPA), asynchronously loaded content and dynamically rendered pages, element attributes and positions may change frequently, resulting in high positioning failure rates.

[0006] 3. Low efficiency of false judgment processing: When element positioning fails, existing technologies mostly require manual intervention to analyze the cause and repair the script, lack of automated repair and feedback learning mechanism, and cannot achieve closed-loop optimization.

[0007] 4. Lack of dynamically updated rule library: There is no intelligent positioning solution in existing technologies that can dynamically update the rule library and use AI self-learning to automatically optimize positioning strategies based on historical positioning success experience.

[0008] 5. Insufficient use of multi-modal features: Existing technologies do not fully utilize DOM structure, visual features, semantic information and other multi-modal features in element positioning, resulting in insufficient positioning accuracy in complex scenarios.

[0009] Therefore, there is an urgent need for an element positioning method that can automatically detect element positioning abnormalities, intelligently generate repair rules, dynamically update feature libraries, and has self-learning capabilities to improve the stability and efficiency of automated testing and reduce maintenance costs. SUMMARY

[0010] To solve the technical problems of strong dependence on static rules, poor adaptability to dynamic content, low misjudgment processing efficiency, lack of dynamically updated rule library, and intelligent positioning solutions using AI self-learning in automated testing or web / application element positioning, achieve automatic repair and learning capabilities of element positioning, improve adaptability to dynamic content, reduce the need for human intervention, improve positioning accuracy, and provide structured failure reports, the present application provides an intelligent element positioning method based on AI and dynamic feature library, comprising the following steps:

[0011] S1, Abnormality detection and abnormal data capture: Perform element positioning operations through automated scripts. If the positioning fails beyond the preset timeout, capture abnormal data and record the structured snapshot of the current page DOM tree.

[0012] S2, Multi-modal feature extraction: Perform structured feature extraction on the captured abnormal data, parse the DOM tree, and extract the XPath / CSS path and semantic attributes of the target element.

[0013] S3, Dynamic feature library retrieval and matching: First, perform rule retrieval in the dynamic feature library based on the multi-modal feature vector of the current element, calculate the DOM structure similarity, and if there is a rule with a total similarity ≥85% in the dynamic feature library, sort all rules with a total similarity ≥85% by weight from high to low, select the rule with the highest similarity as the correction basis, and perform step S5. If there is no rule with a total similarity ≥85% in the dynamic feature library, perform step S4.

[0014] S4, AI intelligent rule generation: Generate multiple candidate positioning rules based on the multi-modal feature vector and the global structure of the page through a classification model, and sort them by weight. Try the candidate positioning rules one by one, and if successful, stop trying and execute step S5.

[0015] S5, Script correction and real-time feedback: Replace the failed positioning statements in the original script with new rules. If the correction is successful, add the new rule to the dynamic feature library. If the correction fails, return to step S3 or S4 to select other rules for correction until successful correction. If all rules fail, execute step S6.

[0016] S6, Generate structured report: Record complete abnormal data and generate a structured report, and mark possible failure causes.

[0017] As a preferred technical solution, the abnormal data in step S1 further comprises intercepting a screen image and capturing context information; after intercepting the screen image, the target element is located in the region through the coordinate information of the failed operation and image cropping is performed; the captured context information comprises a page URL, a browser and / or a device type, and a user operation sequence.

[0018] As a preferred technical solution, the XPath / CSS path of the target element in step S2 is generated based on an absolute path and a relative path of a DOM level.

[0019] As a preferred technical solution, the semantic attribute of the target element in step S2 comprises element text content, id / class attributes, and adjacent elements.

[0020] As a preferred technical solution, in step S3, the multi-modal feature vector of the current element is subjected to rule retrieval in the dynamic feature library, and a DOM structure similarity is calculated, which is specifically: traversing the rules of the same platform in the dynamic feature library, and calculating a weighted similarity: total similarity = 0.5*DOM structure similarity + 0.5*context similarity; the DOM structure similarity is a path difference calculated based on a tree edit distance algorithm.

[0021] As a preferred technical solution, in step S4, the plurality of candidate positioning rules are generated by a classification model, which is specifically: Top-3 candidate positioning rules are generated by a LightGBM classification model.

[0022] As a preferred technical solution, in step S4, the determination condition of successful positioning is that the element is interactive and the attribute meets the expectation.

[0023] As a preferred technical solution, in step S5, if the new rule is added to the dynamic feature library when the successful correction is made, the initial weight = 1, and the weight of the same rule in the dynamic feature library increases by a when it is successfully used once; in step S5, when the correction fails, the weight decreases by β, and when the weight of a rule in the dynamic feature library is less than a threshold value, the rule is set to a frozen state and no longer participates in matching.

[0024] The application also provides an intelligent element positioning system based on AI and a dynamic feature library for realizing the above method, comprising an abnormality detection and abnormal data capturing module, an intelligent positioning module, a dynamic feature library module, an abnormality processing module, a script correction and feedback module, and a structured report generation module; wherein:

[0025] The abnormality detection and abnormal data capturing module is used for detecting positioning abnormalities and capturing abnormal data.

[0026] The intelligent positioning module is used for multi-modal feature extraction, and extracts XPath / CSS paths and semantic attributes of target elements.

[0027] The dynamic feature library module is used for searching and similarity matching in the dynamic feature library;

[0028] The anomaly processing module is used for generating a matching rule or an AI positioning according to the priority of the failure type for the input multi-modal feature vector and the page global structure;

[0029] The script correction and feedback module is used for replacing the invalid positioning statement in the original script with a new rule and adaptively adjusting the rule in the dynamic feature library;

[0030] The structured report generation module is used for recording complete anomaly data and generating a structured report.

[0031] As a preferred technical solution, the intelligent positioning module further comprises an anchor point rule definition module, which is used for relative anchor point positioning, selects a nearby high-stability element as a reference point based on page layout stability analysis, and dynamically generates a relative path.

[0032] The present application realizes a "detection → matching → self-repairing → learning" closed loop through a three-stage architecture of multi-modal feature extraction, dynamic feature library weight accumulation and AI anchor point rule generation. Specifically, the beneficial effects of the present application are as follows:

[0033] 1. The automatic repair and learning ability of element positioning is realized, and when the original positioning rule is invalid, the system can automatically generate a new rule and verify it;

[0034] 2. The adaptability to dynamic content is improved, and through multi-modal feature extraction and relative anchor point positioning, dynamic generated elements and complex UI interaction scenarios are effectively handled;

[0035] 3. The need for manual intervention is reduced, and through the AI intelligent rule generation and feedback learning mechanism, the maintenance cost is reduced;

[0036] 4. The positioning accuracy is improved through the weight management mechanism, the weight of the successful rule is increased, the weight of the failed rule is reduced, and the self-optimization of the system is realized;

[0037] 5. A structured failure report is provided, which can provide valuable information for manual intervention even when all rule attempts fail. BRIEF DESCRIPTION OF DRAWINGS

[0038] Figure 1 The flowchart of the embodiment of the present application, an intelligent element positioning method based on AI and dynamic feature library. DETAILED DESCRIPTION

[0039] In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the following will clearly and completely describe the technical solutions in the embodiments of the present application with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort belong to the scope of the present application.

[0040] The terms used in the embodiments of the present application are only for the purpose of describing particular embodiments and are not intended to limit the present application. The singular forms "a", "said" and "the" used in the embodiments of the present application and the appended claims are also intended to include the plural forms, unless the context clearly indicates otherwise. "Plural" generally includes at least two, but does not exclude the case of including at least one.

[0041] It should be understood that the term "and / or" used herein is only to describe the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B, which means that there are three cases of A alone, A and B together, and B alone. In addition, the character " / " in this paper generally represents that the front and rear associated objects are a "or" relationship.

[0042] Depending on the context, the word "if" as used herein can be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting". Similarly, depending on the context, the phrase "if it is determined" or "if (a stated condition or event) is detected" can be interpreted as "when it is determined" or "in response to determining" or "when (a stated condition or event) is detected" or "in response to detecting (a stated condition or event)".

[0043] It should also be noted that the terms "comprising", "including", or any other variant thereof are intended to cover non-exclusive inclusion, so that a product or system including a series of elements not only includes those elements, but also includes other elements not explicitly listed, or further includes elements inherent to such product or system. Without more limitations, the element defined by the sentence "including a" does not exclude the presence of other identical elements in the product or system including the element.

[0044] As Figure 1 A specific implementation of an intelligent element positioning method based on AI and dynamic feature library of the present application is shown. The intelligent element positioning method based on AI and dynamic feature library of the present application includes the following steps:

[0045] S1, Abnormality detection and abnormal data capture: Perform element positioning operation through automated script, if exceeding the preset timeout, determine as positioning failure, perform abnormal data capture, record the structured snapshot of the current page DOM tree. The trigger condition of abnormality detection is: when performing element positioning operation (such as click() or find_element()) through automated script, if exceeding the preset timeout (such as 5 seconds) and not finding the target element, the system determines as positioning failure. After positioning failure, perform abnormal data capture, record the structured snapshot of the current page DOM tree, the structured snapshot of the current page DOM tree includes parent-child node hierarchy, element attribute. In this embodiment, the abnormal data further includes intercepting screen image and capturing context information; after intercepting the screen image, locate the target element area through the coordinate information of the failed operation and perform image cropping; the captured context information includes page URL, browser and / or device type, user operation sequence (such as "login -> search -> click product").

[0046] In this embodiment, the automated script is a web automated test script based on Selenium framework, used to simulate user's interactive operation on the webpage. The preset timeout is set to 5 seconds (in actual application, the timeout can be adjusted according to needs), that is, when the script tries to locate a certain element for more than 5 seconds and still fails, the system will trigger the abnormality detection mechanism. The abnormality detection mechanism is realized by monitoring the script execution state, when capturing NoSuchElementException or TimeoutException exception, immediately start the abnormal data capture process.

[0047] During the abnormal data capture process, the system will call the built-in API of the browser driver to obtain the complete DOM tree of the current page, and serialize it into JSON format for saving. The structured snapshot of the DOM tree includes the tag name, attribute, hierarchical relationship and other information of all elements in the page, which constitutes the basic data for subsequent analysis.

[0048] In addition to recording the structured snapshot of the DOM tree, the system will also intercept the screen image of the current page, and locate the target element area according to the coordinate information of the failed operation, and perform image cropping on the area, so as to facilitate subsequent image analysis. The system uses the screenshot_as_png method of Selenium to obtain the screenshot of the entire page, in order to improve the accuracy of subsequent analysis, the system will locate the target element area through the coordinate information of the failed operation, and perform image cropping. Then according to the coordinates of the last mouse operation, combined with the page scroll position, calculate the approximate position of the target element in the screenshot, and perform image cropping. The image cropping adopts intelligent area recognition algorithm, instead of simple fixed size cropping. The algorithm steps are as follows:

[0049] 1. Initialize a 200x200 pixel cropping area centered on the target coordinate;

[0050] 2. Identify the element boundaries within the cropping area using edge detection algorithms;

[0051] 3. Expand the cropping area until it fully contains the target element and its directly related visual context;

[0052] 4. If the identified element area is too large (exceeding 500x500 pixels), keep the original cropping area unchanged.

[0053] After cropping, the system will preprocess the image, including adjusting brightness and contrast, removing noise, and marking the target area, to facilitate subsequent image analysis.

[0054] At the same time, the system will also capture context information, including page URL, browser type and version, device type, and user operation sequence. The page URL not only includes the complete URL of the current page, but also includes the parsing results of the URL, such as domain name, path, query parameters, etc.; browser information includes browser type, version number, rendering engine version, etc.; device type information includes operating system, device model, screen resolution, etc. The user operation sequence is a timestamp-ordered operation list that records all the interactive operations performed by the user in the current session. Each operation record contains the following information: 1. Timestamp: the exact time of operation execution; 2. Operation type: such as click, input, scroll, hover, etc.; 3. Target element: a brief description of the target element of the operation; 4. Operation parameters: such as input text content, scroll distance, etc.; 5. Operation result: whether the operation is successfully completed.

[0055] The page URL is obtained through driver.current_url; the browser type and version are obtained through the driver.capabilities attribute; the device type is obtained by parsing the User-Agent string; and the user operation sequence is an ordered list formed by recording all interactive operations (such as clicks, inputs, scrolls, etc.) and their timestamps during script execution. These detailed context information is crucial for understanding the environmental background of element positioning failure, and helps the system to make more accurate analysis and repair. For example, by analyzing the user operation sequence, the system can identify key operations that may cause changes in page state; through browser and device information, the system can identify compatibility issues that may be related to specific platforms.

[0056] S2, Multimodal feature extraction: Structured feature extraction on the captured abnormal data, parse DOM tree, extract XPath / CSS path and semantic attributes of target elements. In this embodiment, the XPath / CSS path of the target element is generated based on the DOM hierarchy to generate absolute path and relative path; the semantic attributes of the target element include element text content, id / class attributes, adjacent elements (such as the placeholder text of the left input box). For example, the button element may extract {"text": "Submit", "class": "btn-primary", "neighbor_text": "Username input box"}.

[0057] In this embodiment, in step S2, the system first parses the captured DOM tree in depth, and constructs the hierarchical relationship diagram of the elements. The parsing process uses a depth-first search algorithm, starting from the root node to traverse the entire DOM tree, assigns a unique identifier to each node, and records its parent-child relationship.

[0058] For target elements, the system generates absolute paths and relative paths based on the DOM hierarchy. The absolute path is the complete path from the HTML root node to the target element, which provides the precise position of the element in the DOM tree. The absolute path types generated by the system include:

[0059] 1. Complete index path: such as " / html / body / div[6] / section / div[7] / button[8]", which contains the index of each node in the same level; 2. Label attribute mixed path: such as " / html / body / div[@id='main'] / section / div[@class='controls'] / button", which combines label name and key attributes; 3. Complete CSS selector: such as "html>body>div:nth-child(2)>section>div:first-child>button:nth-child(3)"; The relative path is a partial path from a unique ancestor node to the target element, which has better adaptability when the page structure changes.

[0060] The system generates relative path types, including: 1. ID anchor relative path: such as " / / div[@id='main-container'] / / button[@class='submit']", starting from the nearest ancestor element with an ID; 2. Feature node relative path: such as " / / form[contains(@action,'login')] / / input[@type='password']", starting from the root node with a distinctive feature; 3. Relative CSS selector: such as "main-container.form-groupbutton.submit". The path generation process uses a bottom-up analysis method, first determining the characteristics of the target element itself, and then analyzing its ancestor nodes level by level, selecting the most distinctive representation for each level.

[0061] The recognition score is based on the following factors:

[0062] 1. Whether the node has a unique ID (highest recognition);

[0063] 2. Whether the node has a unique name or other key attributes;

[0064] 3. Whether the node's tag name and class name combination is unique among siblings;

[0065] 4. The node's index position among siblings.

[0066] For each level, the system calculates the recognition score of different representations and selects the highest scoring one. For example, if a div element has a unique ID, the system will prefer to use "div[@id='unique-id']" instead of "div[6]".

[0067] In addition, the system also analyzes the overall structure of the page and identifies possible dynamic generation patterns. For example, if it finds a large number of ID naming patterns such as "item-1", "item-2" in the page, the system will avoid directly using these IDs that may change dynamically, and instead look for more stable features or use pattern matching expressions, such as "div[starts-with(@id,'item-')]".

[0068] By generating multiple types of absolute paths and relative paths, the system greatly improves the success rate of element positioning. In actual application, the system will dynamically adjust the priority of different path types according to the characteristics of the page and historical success rate, and select the most suitable positioning strategy for the current scenario. The system will generate paths in both XPath format and CSS selector format to increase the flexibility of subsequent positioning.

[0069] The XPath path generation algorithm analyzes the hierarchical relationship of elements and attempts to generate different types of XPath expressions using various strategies:

[0070] 1. Index-based XPath: such as " / html / body / div[6] / button[7]";

[0071] 2. Attribute-based XPath: such as " / / button[@id='submit-btn']";

[0072] 3. XPath based on text content: such as " / / button[text()='Submit']";

[0073] 4. XPath based on partial attribute matching: such as " / / button[contains(@class,'btn-primary')]".

[0074] CSS selector generation also employs multiple strategies:

[0075] 1. ID selector: such as "submit-btn";

[0076] 2. Class selectors: such as ".btn-primary";

[0077] 3. Attribute selectors: such as "button[type='submit']";

[0078] 4. Combination selectors: such as "main-form .form-group button".

[0079] In addition to path information, the system also extracts the semantic attributes of the target element, including the element's text content, id / class attributes, and neighboring elements. The element's text content refers to the visible text within the element, obtained through the `element.text` property; the id / class attributes are the element's unique identifier and style class name, obtained through `element.get_attribute('id')` and `element.get_attribute('class')`; neighboring elements refer to sibling nodes or parent-child nodes adjacent to the target element in the DOM tree, and information about these elements can serve as reference points for auxiliary positioning.

[0080] The following strategy is used to extract neighboring elements:

[0081] 1. Extract the parent element information of the target element;

[0082] 2. Extract the preceding and following sibling elements of the target element;

[0083] 3. Extract the sub-elements of the target element (if any).

[0084] For each neighboring element, the system also extracts its tag name, attributes, and text content, etc. to build a local context description. These semantic properties together form the feature fingerprint of the target element, which helps in fuzzy matching and intelligent positioning when the page structure changes.

[0085] S3, dynamic feature library retrieval and matching: first, the multi-modal feature vector of the current element is retrieved in the dynamic feature library, the DOM structure similarity is calculated, if there is a rule with total similarity ≥ 85% in the dynamic feature library, all rules with total similarity ≥ 85% are sorted by weight from high to low, the rule with the highest similarity is selected as the correction basis, and step S5 is performed; if there is no rule with total similarity ≥ 85% in the dynamic feature library, step S4 is performed. In this embodiment, "the multi-modal feature vector of the current element is retrieved in the dynamic feature library, and the DOM structure similarity is calculated" specifically refers to: traversing the rules of the same platform (such as Web / iOS) in the dynamic feature library, and calculating the weighted similarity: total similarity = 0.5*DOM structure similarity + 0.5*context similarity; the DOM structure similarity is the path difference calculated based on the tree edit distance (Tree Edit Distance) algorithm. Specifically:

[0086] In this embodiment, in the step S3, dynamic feature library retrieval and matching step, the system first converts the multi-modal features extracted in step S2 into a feature vector. The feature vector contains the following dimensions:

[0087] 1. DOM structure features: including element tag name, hierarchical depth, number of sub-elements, etc.

[0088] 2. Attribute features: including id, class, name, type, etc. attribute values;

[0089] 3. Text features: text content and its length, language features of the element;

[0090] 4. Position features: relative position and size of the element in the page;

[0091] 5. Context features: including key information of parent elements and sibling elements.

[0092] After the feature vector is constructed, the system starts the rule retrieval process in the dynamic feature library. The dynamic feature library is a structured database that stores historical successful positioning rules and their corresponding feature vectors, weights, and usage statistics. The retrieval process first filters out the rule subset that matches the current page platform (such as Web, Android, iOS), and then performs similarity calculation in this subset.

[0093] The system traverses the rules of the same platform in the dynamic feature library and calculates the weighted similarity. The total similarity calculation formula is: Total Similarity = 0.5 * DOM Structure Similarity + 0.5 * Context Similarity.

[0094] The DOM structure similarity is based on the tree edit distance algorithm to calculate the path difference. The tree edit distance refers to the minimum number of editing operations (inserting, deleting, replacing nodes) required to convert one tree into another. This embodiment uses an improved Zhang-Shasha algorithm to realize this calculation, which optimizes the traditional tree edit distance algorithm and improves the calculation efficiency.

[0095] Specifically, this embodiment parses the XPath path of the target element and the XPath path of the rule in the feature library into tree structures respectively, and each node contains tag name, attribute, and index information. Then the edit distance between the two trees is calculated, and the cost of editing operation is defined as follows:

[0096] 1. Insert node: cost is 1;

[0097] 2. Delete node: cost is 1;

[0098] 3. Replace node: cost is dynamically calculated according to node similarity, ranging from 0 to 1;

[0099] - If the tag name is the same: the base cost is 0;

[0100] - If the tag name is different: the base cost is 0.6;

[0101] - Attribute difference: each different key attribute increases the cost by 0.1;

[0102] - Index difference: a difference of 1 in index increases the cost by 0.05, with a maximum increase of 0.2;

[0103] After the edit distance is calculated, the DOM structure similarity is calculated by the following formula:

[0104] DOM Structure Similarity = 1 - (Edit Distance / max(Tree 1 Node Number, Tree 2 Node Number)).

[0105] Context similarity considers the attributes of elements, text content, and neighboring elements, and uses weighted cosine similarity calculation. The system converts various features of elements into feature vectors, and then calculates the cosine similarity between the feature vectors of the current element and the rule elements in the feature library. The feature vectors include the following dimensions:

[0106] 1. Element tag name (one-hot encoding);

[0107] 2. Element id / class attribute (TF-IDF vector);

[0108] 3. Element text content (bag-of-words model vector);

[0109] 4. Element other attributes (key-value pair vector);

[0110] 5. Neighboring element features (compressed representation vector);

[0111] 6. Page URL features (domain name and path encoding vector);

[0112] 7. User operation sequence features (operation type and order encoding vector).

[0113] For different types of features, the system assigns different weights:

[0114] 1. The id attribute weight is 0.3;

[0115] 2. The class attribute weight is 0.2;

[0116] 3. The text content weight is 0.3;

[0117] 4. The other attribute weight is 0.1;

[0118] 5. The neighboring element feature weight is 0.1.

[0119] The calculation formula of context similarity is:

[0120] Context similarity = Σ (feature i weight * feature i cosine similarity) / Σ feature i weight

[0121] To improve retrieval efficiency, the system uses a multi-level index structure:

[0122] 1. First-level index: platform type;

[0123] 2. Second-level index: page URL pattern;

[0124] 3. Third-level index: element tag name;

[0125] 4. Fourth-level index: key attribute hash.

[0126] This multi-level index structure greatly reduces the number of candidate rules that need to be calculated for similarity, improving retrieval speed. In practical applications, for a large feature library containing hundreds of thousands of rules, the system can complete retrieval and similarity calculation in milliseconds.

[0127] After the calculation is completed, the system checks whether there are rules with a total similarity ≥ 85%. If there are, all rules with a total similarity ≥ 85% are sorted by weight from high to low. The weight is a dynamically adjusted value that reflects the historical success rate and applicability of the rule. After sorting, the system selects the rule with the highest similarity as the basis for correction and enters step S5.

[0128] If there are no rules in the dynamic feature library with a total similarity ≥ 85%, it means that the features of the current element are significantly different from the historical rules, and a new positioning rule needs to be generated through AI intelligence. Therefore, the system enters step S4.

[0129] S4, AI intelligent rule generation: based on the multi-modal feature vector and the page global structure, a plurality of candidate positioning rules are generated through a classification model, and are sorted by weight from high to low. The candidate positioning rules are tried in turn, and if successful positioning is achieved, the attempt is stopped and step S5 is executed. In this embodiment, during AI intelligent rule generation, the model input is a multi-modal feature vector and a page global structure (such as a page layout skeleton graph); then a Top-3 candidate positioning rule is generated through a LightGBM classification model. For example:

[0130] Relative 1. XPath: / / div[contains(@class, "header")] / / button[text()="Submit"].

[0131] 2. Image coordinates: (x = 320, y = 480) combined with OCR text verification.

[0132] 3. CSS selector:.form-container>.btn:last-child.

[0133] Then rule verification and optimization are performed: candidate rules are tried in turn, and if successful positioning is achieved, the attempt is immediately stopped and the execution result is recorded. The success determination condition is that the element is interactive (clickable / enterable) and the attribute meets the expectation (such as displayed == true).

[0134] In this embodiment, in the step S4, AI intelligent rule generation step, the system uses a pre-trained machine learning model to generate positioning rules suitable for the current scenario based on the multi-modal feature vector and the page global structure. The system uses a LightGBM classification model to generate Top-3 candidate positioning rules.

[0135] LightGBM is a high-efficiency machine learning algorithm based on gradient boosting decision trees, with advantages of fast training speed, low memory occupation, and high accuracy. In this system, the LightGBM model receives the following input features:

[0136] 1. Basic features of elements such as tag name, attributes, and text content;

[0137] 2. Position and hierarchy information of elements in the DOM tree;

[0138] 3. Global structural features of the page, such as the depth, width, and node density of the DOM tree;

[0139] 4. Pattern statistics of historical successful positioning rules.

[0140] The output of the model is the success probability score of different positioning strategies, including:

[0141] 1. Success probability of ID positioning strategy;

[0142] 2. Success probability of CSS selector positioning strategy;

[0143] 3. Success probability of XPath positioning strategy;

[0144] 4. Success probability of text content positioning strategy;

[0145] 5. Success probability of relative positioning strategy.

[0146] Based on these probability scores, the system generates specific positioning rules for each positioning strategy. For example, if the success probability of the ID positioning strategy is the highest, the system will generate positioning rules based on element ID; if the success probability of the relative positioning strategy is high, the system will generate relative positioning rules based on adjacent stable elements.

[0147] For each positioning strategy, the system generates multiple candidate rules and calculates the weight of the rules based on the following factors:

[0148] 1. Success probability score of the positioning strategy;

[0149] 2. Specificity of the rule (the more specific the rule, the higher the weight);

[0150] 3. Robustness of the rule (resistance to changes in page structure);

[0151] 4. Success rate statistics of similar historical rules.

[0152] After the weight calculation, the system sorts all candidate positioning rules by weight from high to low and selects the top-3 rules as the final candidates. These candidate rules usually include different types of positioning strategies to increase the likelihood of successful positioning.

[0153] The system tries these candidate positioning rules in turn. For each rule, the system performs the positioning operation on the current page and verifies whether the positioning result meets the expectations. The success criterion for positioning is that the element is interactive and the attributes meet the expectations. In this embodiment, the system adopts a strict multi-verification mechanism for the success criterion of positioning to ensure that the positioned element not only exists in the page but also can be normally interacted and meets the expected attribute characteristics. Specifically, the system checks the following points:

[0154] 1. Whether the positioned element exists and is visible;

[0155] 2. Whether the positioned element is interactive (such as clickable, inputtable, etc.);

[0156] 3. Whether the key attributes of the positioned element match the target element.

[0157] Element existence verification is the most basic criterion. The system checks the following points:

[0158] 1. Whether the element exists in the DOM tree;

[0159] 2. Whether the element is visible in the viewport (not blocked by other elements);

[0160] 3. Whether the element has a visible size (width and height greater than 0);

[0161] 4. Whether the element is not hidden by CSS (display:none or visibility:hidden).

[0162] Element interactivity verification is to determine whether the element can respond to user operations. The system performs different verifications according to the element type:

[0163] 1. For clickable elements such as buttons, links, etc.: verify whether the element can receive click events;

[0164] 2. For form elements such as input boxes, text areas, etc.: verify whether the element can receive input;

[0165] 3. For selection elements such as drop-down menus, checkboxes, etc.: verify whether the element can change the selected state;

[0166] 4. For drag-and-drop elements: verify whether the element can respond to drag-and-drop operations.

[0167] Interactivity verification uses JavaScript injection. The system injects a test script into the page to simulate user interaction and capture element responses. For example, for a button element, the system injects the following script:

[0168] ```javascript

[0169] function checkClickable(element) {

[0170] / / Check if the element is disabled

[0171] if (element.disabled || element.getAttribute('disabled') !== null){

[0172] return false;

[0173] }

[0174] / / Check if the element is covered

[0175] var rect = element.getBoundingClientRect();

[0176] var centerX = rect.left + rect.width / 2;

[0177] var centerY = rect.top + rect.height / 2;

[0178] var elementAtPoint = document.elementFromPoint(centerX, centerY);

[0179] / / Check if the element at the mouse pointer position is the target element or its child element.

[0180] return element === elementAtPoint || element.contains(elementAtPoint);

[0181] }

[0182] ```

[0183] Element attribute conformance verification determines whether the located element matches the expected target. The system compares the following key attributes:

[0184] 1. Tag name: Whether the HTML tag of the element matches the expectation;

[0185] 2. Element type: Whether the type attribute of the input element matches the expectation;

[0186] 3. Key attributes: Whether the key attributes such as id, name, class of the element match the expectation;

[0187] 4. Text content: Whether the visible text of the element matches the expectation;

[0188] 5. Element role: Whether the ARIA role of the element matches the expected function.

[0189] Attribute compliance verification uses a fuzzy matching method, allowing partial attribute changes. The system assigns different weights to different attributes and calculates the attribute matching degree:

[0190] 1. id attribute weight: 0.3;

[0191] 2. Tag name weight: 0.2;

[0192] 3. Text content weight: 0.3;

[0193] 4. class attribute weight: 0.1;

[0194] 5. Other attribute weight: 0.1.

[0195] The calculation formula of attribute matching degree is:

[0196] Attribute matching degree = Σ (attribute i weight * attribute i matching degree)

[0197] Where attribute i matching degree is 0 (not matching) or 1 (matching). For text content, the system uses a string similarity algorithm to calculate the matching degree, allowing partial text changes.

[0198] Only when the element passes the existence verification, the interactivity verification, and the attribute matching degree ≥ 0.8, the system determines that it is successfully located. This strict multiple verification mechanism ensures the accuracy and reliability of the positioning result, avoiding the failure of automated scripts caused by false positioning.

[0199] If a rule successfully locates an element that meets the expectations, the system will immediately stop trying other rules and use the successful rule as the basis for correction, entering step S5. If all candidate rules cannot successfully locate, the system will adjust the feature weights and generate new candidate rules for trial until a successful rule is found or the number of attempts reaches the upper limit.

[0200] S5, script correction and real-time feedback: replace the failed positioning statement in the original script with the new rule, if the correction is successful, add the new rule to the dynamic feature library, if the correction fails, return to step S3 or S4 to select other rules for correction until successful correction, if all rules fail, execute step S6. In this implementation, step S5 first performs dynamic script replacement, replacing the failed positioning statement in the original script with a newly generated rule (such as replacing the old XPath with a relative path). For example:

[0201] # Original statement (failed)

[0202] driver.find_element(By.XPATH, ' / / button[@id="submit_btn"]')

[0203] # After correction

[0204] driver.find_element(By.XPATH, ' / / div[@class="modal-footer"] / button[1]')

[0205] Then perform the feedback learning mechanism, if the new rule is successfully added to the dynamic feature library, the initial weight = 1, and the weight of the same rule in the dynamic feature library increases by α (which can be set as needed, for example, α = 1.0) for each successful use; When the correction fails in step S5, the weight decreases by β (which can be set as needed, for example, β = 0.5), and when the weight of a rule in the dynamic feature library is less than a threshold value (which can be set as needed, for example, 0.2), the rule is set to a frozen state and no longer participates in matching.

[0206] In this embodiment, in the script correction and real-time feedback step S5, the system first analyzes the failed positioning statement in the original automation script to determine its position and context in the script. The failed positioning statement is usually an element selector expression, such as driver.find_element(By.XPATH, " / / button[@id='submit']").

[0207] The system generates a new positioning statement based on the successful positioning rule obtained in step S3 or S4. The new positioning statement maintains the same API call format as the original statement, but uses a new selector expression. For example, if the new positioning rule uses the CSS selector ".btn-primary[type='submit']", the generated new positioning statement is driver.find_element(By.CSS_SELECTOR, ".btn-primary[type='submit']").

[0208] After generating the new locating statement, the system replaces the invalid locating statement in the copy of the original script and executes the modified script snippet in the isolated environment to verify whether the new locating rule can successfully locate the target element. The verification process includes the following steps:

[0209] 1. Execute the new locating statement in the current page state;

[0210] 2. Check whether the locating result meets the expectations (element exists, is visible, and is interactive);

[0211] 3. Perform the intended operation (such as clicking or inputting) and verify whether the operation is successfully completed.

[0212] If the verification is successful, it indicates that the correction is valid, and the system adds the new rule to the dynamic feature library. During the addition process, the system assigns an initial weight = 1 to the new rule and records the detailed information of the rule, including:

[0213] 1. Rule type (XPath, CSS selector, etc.);

[0214] 2. Rule expression;

[0215] 3. Feature vector of the target element;

[0216] 4. Page context information;

[0217] 5. Creation time and last use time.

[0218] The weight of the same rule in the dynamic feature library increases by α (default α = 0.1) each time it is successfully used. This dynamic adjustment mechanism allows frequently successful rules to obtain higher weights, thereby obtaining higher priority in future matching processes.

[0219] If the verification fails, it indicates that the selected rule cannot successfully correct the script, and the system decides to return to step S3 or S4 based on the failure reason. If the failure reason is related to the rule itself (such as selector syntax error), the system returns to step S4 to generate a new candidate rule; if the failure reason is related to the page state (such as page structure change), the system returns to step S3 to re-perform feature library matching.

[0220] When the correction fails, the system reduces the weight of the corresponding rule by β (default β = 0.5). When the weight of a rule in the dynamic feature library is less than the threshold value (default threshold = 0.2), the system freezes the rule and no longer participates in matching to avoid repeatedly attempting invalid rules.

[0221] In this embodiment, the system implements an adaptive rule weight adjustment mechanism, enabling the dynamic feature library to continuously optimize and evolve, improving the accuracy and adaptability of positioning rules.

[0222] When the system successfully corrects the element positioning problem and adds a new rule to the dynamic feature library, it assigns an initial weight = 1 to the new rule. This initial value represents the basic credibility of the rule, neither too high nor too low, giving the new rule a fair starting point.

[0223] The weight of the same rule in the dynamic feature library increases by α each time it is successfully used. By default, α = 1.0, which means that each time a rule is successfully applied, its weight increases by 1. This incremental weight adjustment ensures that frequently successful rules gradually gain higher weights, thereby gaining higher priority in future matching processes.

[0224] The weight increase is not unlimited and can be set according to actual needs, such as a weight upper limit of 5.0. When the rule weight reaches the upper limit, it will no longer increase, but the last use time of the rule will be updated to reflect its continued active state.

[0225] When the correction fails, the weight decreases by β. By default, β = 0.5, which means that each time a rule fails to be applied, its weight decreases by 0.5.

[0226] When the weight of a rule in the dynamic feature library is less than the threshold value, the rule is placed in a frozen state and no longer participates in matching. The default threshold value is 0.3, and when the rule weight is lower than this value, the system considers that the rule is no longer suitable for the current page structure and changes its state from "active" to "frozen".

[0227] Frozen rules are not immediately deleted from the feature library, but are retained for a certain period of time (which can be adjusted according to actual circumstances, with a default of 30 days) and do not participate in matching during this period. This design takes into account the possibility that a page may temporarily change and then return to its original state, avoiding the permanent loss of valuable rules.

[0228] This embodiment also implements a "thawing" mechanism for rules. When the page structure changes significantly, causing the success rate of active rules to drop significantly, the system will attempt to thaw some historical rules, giving them the opportunity to participate in matching again. The conditions for thawing are:

[0229] 1. The current page URL is highly similar to the URL when the rule was created;

[0230] 2. The DOM structure features of the page have changed significantly compared to when the rule was created;

[0231] 3. The success rate of active rules is lower than the pre-set threshold (default is 30%).

[0232] In addition to the basic weight adjustment mechanism, the system also implements the following advanced features:

[0233] 1. Time decay: The weight of a rule that has not been used for a long time will decay slowly. The decay formula is: New weight = Old weight * (0.9 ^ (Number of days not used / 30));

[0234] 2. Context sensitive: The success rate of the same rule may vary in different contexts (such as different URL patterns). The system records the performance of the rule in various contexts and adjusts its effective weight in specific contexts.

[0235] 3. Batch Adjustment: When a major update occurs to the page structure, the system will detect that multiple rules are invalid at the same time. At this time, the batch adjustment mechanism will be triggered to temporarily reduce the penalty coefficient of all related rules and prevent valuable rules from being frozen prematurely.

[0236] 4. Rule Clustering: The system will cluster rules with similar functions. When a rule fails, it will check the performance of other rules in the same cluster to determine whether it is a local problem or a change in the overall page structure.

[0237] Through this complex and sophisticated weight adjustment mechanism, the dynamic feature library can continuously optimize itself, retain high-quality positioning rules, eliminate invalid rules, and adapt to dynamic changes in page structure, providing continuous and effective element positioning services.

[0238] The system will continuously try different rules to make corrections until a successful solution is found. If all possible rules fail, the system will execute step S6 to generate a structured report.

[0239] S6. Generate a structured report: Record complete exception data and generate a structured report, noting possible reasons for failure. If all rule attempts fail, the system executes:

[0240] Record the complete error context (including DOM, screenshots, and logs).

[0241] Generate a structured report (JSON / HTML format), noting possible reasons for failure:

[0242] {

[0243] "error_code": "E404",

[0244] "suspected_cause": "Dynamic element ID change + inconspicuous visual features",

[0245] "suggested_action": "Manually define static anchor points"

[0246] }

[0247] In this embodiment, in step S6, the structured report generation step, the system integrates the information collected in the previous steps to generate a detailed exception analysis report. The report is in JSON format and includes the following main parts:

[0248] 1. Basic information section:

[0249] - Time of exception occurrence;

[0250] - Page URL;

[0251] - Browser type and version;

[0252] - Device type;

[0253] - Script execution environment.

[0254] 2. Exception details section:

[0255] - Exception type (e.g. NoSuchElementException, TimeoutException, etc.);

[0256] - Exception stack information;

[0257] - Failing locating statement;

[0258] - Expected operation type (click, input, etc.).

[0259] 3. Page analysis section:

[0260] - DOM tree structure summary;

[0261] - Target element feature vector;

[0262] - Page screenshot path;

[0263] - Element area screenshot path.

[0264] 4. Repair attempt section:

[0265] - List of attempted locating rules;

[0266] - Verification result of each rule;

[0267] - Final selected repair solution (if any).

[0268] 5. Failure reason analysis section:

[0269] - Possible failure reason categories;

[0270] - Detailed description of failure cause;

[0271] - Confidence score.

[0272] The system automatically annotates possible failure causes by analyzing the collected data. The analysis of failure causes is based on the following aspects:

[0273] 1. Whether the element exists but is not visible (e.g., blocked by other elements, in a collapsed state, etc.);

[0274] 2. Whether the element has been removed from the page;

[0275] 3. Whether the properties of the element have changed;

[0276] 4. Whether there has been a significant change in the page structure;

[0277] 5. Whether there are dynamic loading or asynchronous updating issues;

[0278] 6. Whether there are special structures such as iframes or shadow DOM.

[0279] For each possible failure cause, the system will give a confidence score (0-100%) and provide detailed evidence to support it. For example, if the system determines that element property changes are the most likely cause, it will list the original and current properties and the time point of the change (if traceable).

[0280] After the structured report is generated, the system will save it to a designated location and optionally send a notification to relevant personnel. The information in the report not only helps developers understand and solve the current problem, but also provides valuable data for subsequent optimization of automated test scripts and improvement of AI models.

[0281] The embodiment also provides an AI and dynamic feature library based intelligent element positioning system for implementing the above AI and dynamic feature library based intelligent element positioning method, comprising an abnormality detection and abnormality data capturing module, an intelligent positioning module, a dynamic feature library module, an abnormality processing module, a script correction and feedback module, and a structured report generation module; wherein the abnormality detection and abnormality data capturing module is configured to detect positioning abnormalities and capture abnormality data; the intelligent positioning module is configured to extract multi-modal features, extract XPath / CSS paths and semantic attributes of target elements; the dynamic feature library module is configured to search and perform similarity matching in the dynamic feature library; the abnormality processing module is configured to generate matching rules or AI positioning according to the input multi-modal feature vector and page global structure, and select priorities according to failure types; the script correction and feedback module is configured to replace invalid positioning statements in original scripts with new rules, and adaptively adjust rules in the dynamic feature library; and the structured report generation module is configured to record complete abnormality data and generate a structured report.

[0282] In the embodiment, the abnormality detection and abnormality data capturing module is an entry component of the system, responsible for monitoring the execution state of the automation script and discovering element positioning abnormalities in a timely manner. The module is designed in an observer mode, and registers an abnormality listener in a script execution engine. When capturing NoSuchElementException, TimeoutException and other element positioning related exceptions, the abnormality processing process is triggered immediately.

[0283] The abnormality detection and abnormality data capturing module comprises three sub-components: an abnormality listener, a data collector and a snapshot generator. The abnormality listener is responsible for intercepting and classifying abnormalities; the data collector is responsible for collecting context information when the abnormality occurs, including page URL, browser type, device information, etc.; and the snapshot generator is responsible for creating a structured snapshot of the page DOM tree and saving a page screenshot.

[0284] During the abnormality data capturing process, the module intercepts screen images and captures context information. After intercepting the screen image, the target element area is located and the image is cropped through the coordinate information of the failed operation. The default cropping area is a 200x200 pixel area centered on the target coordinate, but the system dynamically adjusts the cropping area size according to the page scaling ratio and device resolution. The captured context information includes page URL, browser and / or device type, user operation sequence. The user operation sequence is a timestamp ordered operation list, recording all interactive operations performed by the user in the current session, including operation type, target element and operation parameter information.

[0285] The intelligent positioning module is the core component of the system, responsible for multi-modal feature extraction of captured abnormal data. The module first parses the DOM tree, then extracts the XPath / CSS path and semantic attributes of the target element. The extraction of XPath / CSS path is based on the generation of absolute path and relative path based on DOM hierarchy. The absolute path starts from the HTML root node and contains the complete path to the target element; the relative path is a partial path starting from a root node with a unique identifier.

[0286] The intelligent positioning module also includes an anchor point rule definition module, which is used for relative anchor positioning. Based on the analysis of page layout stability, the module selects nearby high-stability elements as reference points and dynamically generates relative paths. The anchor point rule definition module first analyzes the stability of all elements in the page. The stability score is based on the following factors:

[0287] 1. Whether the element has a unique and stable id or name attribute

[0288] 2. The frequency of location changes in historical versions

[0289] 3. The DOM level depth of the element (the shallower, the more stable)

[0290] 4. The content update frequency of the element

[0291] After scoring, the system selects the element closest to the target element with a stability score ≥ 0.8 as the anchor point. After the anchor point is determined, the system generates a relative path from the anchor point to the target element, such as ". / following-sibling::div[6] / button" or ".. / div[@class='content'] / span". This relative anchor positioning method greatly improves the robustness of element positioning. Even if the page structure changes, as long as the anchor element and the relative structure remain stable, the positioning will still be effective.

[0292] When extracting the semantic attributes of the target element, the intelligent positioning module includes element text content, id / class attributes, and adjacent elements. Element text content refers to the visible text inside the element. The system will extract the complete text and perform natural language processing to identify keywords and semantic features; id / class attributes are the unique identifiers and style class names of the element. The system will analyze the naming patterns and semantic meanings of these attributes; adjacent elements refer to nodes adjacent to the target element in the DOM tree. The system will extract the key features of these elements to construct a local context environment description.

[0293] Dynamic Feature Library Module is the knowledge base component of the system, responsible for storing and managing historically successful positioning rules. This module adopts a distributed database architecture, supporting high concurrency read and write and fast retrieval. Each rule record in the feature library contains the following fields:

[0294] 1. Rule ID: Unique identifier

[0295] 2. Rule Type: Such as XPath, CSS selector, relative anchor point, etc.

[0296] 3. Rule Expression: Specific positioning expression

[0297] 4. Feature Vector: Multimodal features of target element

[0298] 5. Applicable Platform: Such as Web, Android, iOS, etc.

[0299] 6. Weight: Reflects the historical success rate of the rule

[0300] 7. Usage Statistics: Including success times, failure times, last usage time, etc.

[0301] 8. Status: Active, frozen, or abandoned

[0302] The Dynamic Feature Library Module provides efficient retrieval interfaces, supporting similarity search based on feature vectors. The retrieval process first filters out a subset of rules that match the current platform, then calculates the similarity between the current element features and the library rule features. Similarity calculation uses a weighted approach: Total Similarity = 0.5*DOM Structure Similarity + 0.5*Context Similarity. DOM structure similarity is calculated based on tree edit distance algorithm to calculate path difference, and context similarity is calculated based on cosine similarity of feature vectors.

[0303] The exception handling module is the decision component of the system, responsible for generating matching rules or AI positioning strategies based on the input multimodal feature vector and page global structure. This module contains two main sub-components: rule matching engine and AI reasoning engine.

[0304] The rule matching engine is responsible for finding rules with similarity ≥85% in the dynamic feature library. If rules that meet the conditions are found, the system will sort these rules by weight from high to low, and select the rule with the highest similarity as the correction basis.

[0305] The AI reasoning engine is started when rule matching fails, generating Top-3 candidate positioning rules through the LightGBM classification model. Model input includes element feature vector and page global structure features, and output is the success probability score of different positioning strategies. Based on these scores, the system generates specific positioning rules and sorts them by weight.

[0306] The script correction and feedback module is an execution component of the system, responsible for applying new rules to the original script and verifying the correction effect. The module first analyzes the invalid positioning statements in the original script, and then generates replacement statements according to the selected new rules. After the replacement is completed, the system executes the modified script segment in an isolated environment to verify whether the new rule can successfully locate the target element.

[0307] If the verification is successful, the system will add the new rule to the dynamic feature library, and the initial weight = 1. The weight of the same rule in the dynamic feature library increases by a (default a = 0.1) for each successful use; when the correction fails, the weight decreases by b (default b = 0.2). When the weight of a rule in the dynamic feature library is less than the threshold value (default threshold value = 0.3), the system will freeze the rule and no longer participate in matching.

[0308] The structured report generation module is an output component of the system, responsible for recording complete abnormal data and generating structured reports. The report adopts JSON format, including abnormal basic information, details, page analysis, repair attempts and failure reason analysis, etc. The system automatically annotates possible failure reasons by comprehensively analyzing the collected data, and provides confidence scores and evidence support for each reason.

[0309] The application discloses an intelligent element positioning method and system based on AI and a dynamic feature library, and belongs to the technical field of automatic testing. The method comprises the following steps: abnormality detection and abnormal data capture, element positioning operation is performed through an automatic script, a page DOM tree structured snapshot is recorded when it is determined that positioning fails; multi-modal feature extraction, a DOM tree is parsed, and an XPath / CSS path and semantic attributes of a target element are extracted; dynamic feature library retrieval and matching, DOM structure similarity is calculated, and if there is a high-similarity rule, the rule is used as a correction basis; AI intelligent rule generation, a candidate positioning rule is generated through a classification model; script correction and real-time feedback, an invalid positioning statement is replaced, and a dynamic feature library is updated; and a structured report is generated, and abnormal data is recorded. The system comprises an abnormality detection and abnormal data capture module, an intelligent positioning module, a dynamic feature library module, an abnormality processing module, a script correction and feedback module, and a structured report generation module. The application realizes intelligent correction of element positioning in an automatic testing process by combining AI and a dynamic feature library, and improves the stability and efficiency of automatic testing.

[0310] In summary, the above is only the preferred embodiment of the application, and is not intended to limit the scope of the application. Any equivalent changes and modifications made within the scope of the application are within the technical scope of the application.

Claims

1. An AI and dynamic feature library-based intelligent element positioning method, characterized in that: Comprise the following steps: S1, anomaly detection and abnormal data capture: the element positioning operation is performed by an automated script, if the positioning fails beyond the preset timeout, the abnormal data is captured, and the structured snapshot of the current page DOM tree is recorded; S2, multi-modal feature extraction: structured feature extraction is performed on the captured abnormal data, the DOM tree is parsed, and the XPath / CSS path and semantic attributes of the target element are extracted; S3, dynamic feature library retrieval and matching: first, the multi-modal feature vector of the current element is retrieved in the dynamic feature library, the DOM structure similarity is calculated, and the specific steps are as follows: traverse the rules in the dynamic feature library of the same platform, and calculate the weighted similarity: total similarity = 0.5*DOM structure similarity + 0.5*context similarity; DOM structure similarity is the path difference calculated based on tree edit distance algorithm, if there is a rule in the dynamic feature library with total similarity ≥ 85%, then all rules with total similarity ≥ 85% are sorted by weight from high to low, and the rule with the highest similarity is selected as the correction basis, and step S5 is performed; if there is no rule in the dynamic feature library with total similarity ≥ 85%, then step S4 is performed; S4, AI intelligent rule generation: according to the multi-modal feature vector and the global structure of the page, a plurality of candidate positioning rules are generated by a classification model, and are sorted by weight, and the candidate positioning rules are tried in turn, if successful positioning, stop trying, execute step S5; S5, script correction and real-time feedback: replace the failed positioning statement in the original script with the new rule, if the correction is successful, add the new rule to the dynamic feature library, if the correction fails, return to step S3 or S4 to select other rules for correction, until successful correction, if all rules fail, execute step S6; S6, generate a structured report: record the complete abnormal data and generate a structured report, and mark the possible failure causes. 2.The AI and dynamic feature library-based intelligent element positioning method of claim 1, wherein: The abnormal data in step S1 also includes intercepting a screen image and capturing context information; after intercepting the screen image, the target element area is located by the coordinate information of the failed operation and image cropping is performed; the captured context information includes page URL, browser and / or device type, user operation sequence. 3.The AI and dynamic feature library based intelligent element positioning method of claim 1, wherein: In step S2, the XPath / CSS path of the target element is generated based on the DOM level absolute path and relative path.

4. The AI and dynamic feature library-based intelligent element positioning method according to claim 1, characterized in that: In step S2, the semantic attributes of the target element include element text content, id / class attributes, and adjacent elements.

5. The AI and dynamic feature library-based intelligent element positioning method according to claim 1, characterized in that: In step S4, "generating a plurality of candidate positioning rules by a classification model" specifically refers to generating Top-3 candidate positioning rules by a LightGBM classification model.

6. The AI and dynamic feature library-based intelligent element positioning method according to claim 1, characterized in that: In step S4, the successful positioning condition is that the element is interactive and the attributes meet the expectations.

7. The AI and dynamic feature library-based intelligent element positioning method according to claim 1, characterized in that: In step S5, when the new rule is added to the dynamic feature library after successful correction, the initial weight = 1, and the weight of the same rule in the dynamic feature library increases by a when it is successfully used once; in step S5, when the correction fails, the weight decreases by β, and when the weight of a rule in the dynamic feature library is less than the threshold, the rule is set to a frozen state and no longer participates in matching.

8. An AI and dynamic feature library based intelligent element positioning system for implementing the method of claims 1-7. Comprise abnormality detection and abnormal data capture module, intelligent positioning module, dynamic feature library module, abnormality processing module, script correction and feedback module, structured report generation module;Among them: The abnormality detection and abnormal data capture module is used to detect positioning abnormalities and capture abnormal data; The intelligent positioning module is used for multi-modal feature extraction, extracting the XPath / CSS path and semantic attributes of the target element; The dynamic feature library module is used for retrieval and similarity matching in the dynamic feature library; The abnormality processing module is used to generate matching rules or AI positioning according to the input multi-modal feature vector and page global structure, and select priority according to the failure type; The script correction and feedback module is used to replace the invalid positioning statements in the original script with new rules, and to adaptively adjust the rules in the dynamic feature library; The structured report generation module is used to record complete abnormal data and generate structured reports. 9.The AI and dynamic feature library based intelligent element positioning system of claim 8, wherein: The intelligent positioning module further comprises an anchor point rule definition module, which is used for relative anchor point positioning, selects adjacent high stability elements as reference points based on page layout stability analysis, and dynamically generates relative paths.

Images