Script-adaptive element positioning method, apparatus, device, and medium

Abstract

The application relates to the technical field of software testing, and discloses a script-adaptive element positioning method, device, equipment and medium. The script-adaptive element positioning method provided by the application obtains image information by responding to a positioning failure event, calculates a matching score and a semantic alignment score of a candidate element and a historical element, and realizes accurate positioning of a target element based on dynamic fusion of the two-channel scores according to an environment state, effectively overcomes the problem of invalidation of a positioner caused by interface changes, can effectively cope with the problem of invalidation of the positioner caused by the interface changes, realizes high-precision element positioning by fusing visual matching and semantic understanding, and improves the robustness and self-healing capability of a test script.

Description

Technical Field

[0001] This application relates to the field of software testing technology, specifically to a method, apparatus, device, and medium for script adaptive element location. Background Technology

[0002] In UI automation testing techniques, automated test scripts rely on static attributes of front-end elements. If the page structure changes, the original locators can easily become ineffective, causing the test script to break. Text or structural similarity matching methods cannot distinguish between similar elements or elements with identical textual semantics. For example, if multiple "login" tags exist on a page, it's impossible to determine which is the correct login button to be tested, leading to element misidentification and test failure. Summary of the Invention

[0003] This application provides a script-adaptive element positioning method, apparatus, device, and medium to solve the problem of locator failure caused by interface changes.

[0004] In a first aspect, this application provides a script-adaptive element localization method, the method comprising: responding to a test operation instruction's failure to locate a target element in the current image, obtaining image information where the target element is located, the target element including the element located by the attribute locator during the test; obtaining candidate element list information in the image information, and calculating a matching score between the candidate elements in the candidate element list information and corresponding elements in historical images, the corresponding elements being elements identical to the target element in historical images; constructing corresponding semantic vectors based on the candidate elements and the test operation instruction respectively, and calculating a semantic alignment score between the semantic vector of the candidate elements and the semantic vector of the test operation instruction; and locating the target element based on the matching score and the semantic alignment score.

[0005] Based on the above technical means, the accuracy and robustness of positioning are improved, the interruption of test scripts and a large amount of manual intervention caused by page changes are reduced, thereby achieving continuous and stable operation and efficiency improvement of automated testing.

[0006] In one optional implementation, corresponding semantic vectors are constructed based on candidate elements and test operation instructions, respectively, and a semantic alignment score is calculated between the semantic vectors of the candidate elements and the semantic vectors of the test operation instructions. This includes: constructing text semantic vectors and context semantic vectors representing the regions near the candidate elements based on the candidate elements in the candidate element list, wherein the regions near the candidate elements include other elements; constructing operation semantic vectors based on the test operation instructions; and calculating a semantic alignment score based on the text semantic vectors, context semantic vectors, and operation semantic vectors.

[0007] Based on the above technical means, the semantic matching degree between candidate elements and test operation instructions can be reflected more comprehensively and accurately, effectively reducing the risk of misidentification caused by insufficient semantic understanding, significantly improving the accuracy and robustness of target element positioning, and thus ensuring the stability of automated test scripts when page structure changes.

[0008] In one optional implementation, a semantic alignment score is calculated based on text semantic vectors, context semantic vectors, and operation semantic vectors, including: calculating the text semantic similarity score of each candidate element by the similarity between the text semantic vectors and operation semantic vectors; calculating the context matching score of each candidate element by the similarity between the context semantic vectors and element control semantic vectors, where the element control semantic vectors are pre-stored descriptive information used to characterize the shape of the target element; determining the positional prior probability score of each candidate element based on the historical occurrence position of the target element and the current position of each candidate element; and weightedly fusing the text semantic similarity score, context matching score, and positional prior probability score of each candidate element to obtain the semantic alignment score between each candidate element and the test operation command.

[0009] Based on the above technical means, by weighting and fusing the scores obtained from the three dimensions, the matching information of multiple dimensions is integrated, which can obtain a comprehensive and accurate result of the semantic alignment of candidate elements, providing an accurate scoring basis for target element positioning, avoiding the misidentification problem caused by single-dimensional matching, and significantly improving the robustness and accuracy of script adaptive positioning.

[0010] In one optional implementation, obtaining candidate element list information from image information and calculating matching scores between candidate elements in the candidate element list information and corresponding elements in historical images includes: identifying text regions in image information and obtaining candidate elements and coordinate information corresponding to each text region; sorting each candidate element and corresponding coordinate information to obtain candidate element list information; querying historical images and determining whether the historical images contain target local images of the corresponding elements; if they exist, obtaining feature information of the corresponding elements in the target local images and calculating the matching degree between the feature information of the corresponding elements and the feature information of the candidate elements to obtain matching scores; if they do not exist, performing the steps of constructing corresponding semantic vectors based on candidate elements and test operation instructions respectively, and calculating semantic alignment scores between semantic vectors.

[0011] The aforementioned technical methods improve the overall positioning speed. Simultaneously, they prevent the generation of incorrect matching scores when historical records are missing, thereby eliminating interference with subsequent target element positioning and effectively ensuring positioning accuracy.

[0012] In one optional implementation, locating the target element based on the matching score and semantic alignment score includes: determining the test environment state, which includes at least the DOM structure change state and the network latency state; determining the weight coefficients of the matching score and semantic alignment score respectively according to the test environment state; multiplying the matching score and semantic alignment score by their respective weight coefficients and summing them to obtain a comprehensive score; and determining the target element based on the comprehensive score.

[0013] Based on the above technical means, the accuracy and stability of element localization in complex and ever-changing testing environments are significantly improved, and the probability of secondary localization failure is reduced, thereby achieving more intelligent and robust script-adaptive element localization.

[0014] In one optional implementation, the weight coefficients of the matching score and the semantic alignment score are determined according to the test environment state, including: when the amount of change representing the change state of the DOM structure is greater than a preset change threshold, the weight coefficient corresponding to the matching score is reduced and the weight coefficient corresponding to the semantic alignment score is increased; when the network latency is greater than a preset latency threshold, the weight coefficient corresponding to the matching score is increased and the weight coefficient corresponding to the semantic alignment score is reduced.

[0015] Based on the above technical means, the accuracy and success rate of target element localization in various complex and dynamic testing environments are significantly improved, effectively avoiding secondary localization failures caused by environmental changes, thereby improving the robustness and stability of UI automated testing.

[0016] In an optional implementation, the method further includes: saving process information and corresponding image information of successfully locating the target element; generating a positioning strategy for locating the target element based on the process information and image information; and using the positioning strategy to locate the target element when locating it again.

[0017] Based on the aforementioned technical means, the overall positioning capability is gradually optimized, forming an automatic positioning closed loop that does not require repeated manual intervention. This enables the continuous evolution of positioning capabilities, thereby effectively improving the efficiency and stability of repeated positioning.

[0018] Secondly, this application provides a script-adaptive element localization device, comprising: an acquisition module, configured to acquire image information of the target element in response to a test operation instruction's failure to locate a target element in the current image, wherein the target element includes the element located by the attribute locator during the test; a first calculation module, configured to acquire candidate element list information in the image information and calculate a matching score between the candidate element in the candidate element list information and a corresponding element in a historical image, wherein the corresponding element is the same as the target element in the historical image; a second calculation module, configured to construct corresponding semantic vectors based on the candidate element and the test operation instruction respectively, and calculate a semantic alignment score between the semantic vector of the candidate element and the semantic vector of the test operation instruction; and a localization module, configured to locate the target element based on the matching score and the semantic alignment score.

[0019] Thirdly, this application provides an electronic device, including: a memory and a processor, which are communicatively connected to each other. The memory stores computer instructions, and the processor executes the computer instructions to perform the script-adaptive element positioning method of the first aspect or any corresponding embodiment described above.

[0020] Fourthly, this application provides a computer-readable storage medium storing computer instructions for causing a computer to execute the script-adaptive element positioning method of the first aspect or any corresponding embodiment described above.

[0021] Fifthly, this application provides a computer program product, including computer instructions for causing a computer to execute the script-adaptive element positioning method of the first aspect or any corresponding embodiment described above. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the specific embodiments of this application or the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0023] Figure 1 This is a schematic flowchart of a first type of script-adaptive element positioning method according to an embodiment of this application; Figure 2 This is a schematic diagram of a second flowchart of a script-adaptive element positioning method according to an embodiment of this application. Figure 3 This is a schematic diagram of the third process of the script-adaptive element positioning method according to an embodiment of this application; Figure 4 This is a schematic diagram of the fourth flowchart of the script-adaptive element positioning method according to an embodiment of this application; Figure 5 This is a first example flowchart illustrating script adaptation according to an embodiment of this application; Figure 6 This is a second example flowchart illustrating script adaptation according to an embodiment of this application; Figure 7 This is a structural block diagram of a script-adaptive element positioning device according to an embodiment of this application; Figure 8 This is a schematic diagram of the hardware structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0025] It is understood that before using the technical solutions disclosed in the various embodiments of this application, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this application in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0026] The terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0027] In related technologies, UI automated testing heavily relies on the static properties of front-end elements for location. When the page structure changes, the original locators are prone to failure, leading to large-scale interruptions of test scripts and severely impacting test stability. This means that existing self-healing strategies still require significant manual intervention and cannot be continuously optimized.

[0028] To address this, this application proposes a script-adaptive element positioning method. For example... Figure 1 As shown, the method includes: Step S101: In response to the test operation instruction for the failure to locate the target element in the current image, obtain the image information where the target element is located. The target element includes the element located by the attribute locator during the test.

[0029] Test operation instructions refer to commands or sequences used in automated testing to instruct the system to perform specific operations. Examples include clicking a button, entering text, and swiping a page. Their purpose is to clearly define the behavior that the user or test script expects to perform on the interface.

[0030] The target element refers to the specific UI component that the test operation instructions expect to interact with during automated testing. In the event of a location failure, the target element is the element that the attribute locator originally attempted to locate; it is the object that needs to be identified and determined in this adaptive location process.

[0031] Image information refers to the visual data of the current UI interface, usually presented in the form of images or screenshots. Its purpose is to provide a visual context independent of the DOM structure for subsequent element recognition and matching.

[0032] Specifically, when an automated test script fails to locate a target element using attribute locators such as XPath or ID, the system is triggered to execute an adaptive positioning process. At this point, you can manually capture an image of the current screen, or obtain image data of the current screen by calling the application programming interfaces provided by the operating system or browser, as a basis for subsequent analysis. The target element can be defined as the UI element that the attribute locator initially attempted to locate during the test, such as a button, an input field, or a text label.

[0033] Step S102: Obtain the candidate element list information in the image information, and calculate the matching score between the candidate element in the candidate element list information and the corresponding element in the historical image. The corresponding element is the same as the target element in the historical image.

[0034] The candidate element list information refers to the collection of all UI components that could potentially become target elements, extracted from the current image information, along with their related data. Its purpose is to narrow the search scope and provide objects to be evaluated for subsequent matching and semantic analysis.

[0035] The matching score is a numerical value that quantifies the degree of similarity between a candidate element and a corresponding element in a historical image by comparing their visual or structural features. A higher matching score indicates that the candidate element and the historical target element have a higher visual similarity.

[0036] Specifically, after acquiring the current image information, all visible UI components in the image that may be related to the target element can be manually identified. The visual areas and basic attributes of these components, such as text content and approximate shape, are recorded to form a candidate element list. Simultaneously, image fragments or visual features of the target element from previous successful location attempts can be pre-stored. Then, through pixel comparison or image hashing algorithms, the visual similarity between the current candidate element and the corresponding element in historical images—that is, the same target element in history—is calculated to obtain a matching score.

[0037] Step S103: Construct corresponding semantic vectors based on candidate elements and test operation instructions respectively, and calculate the semantic alignment score between the semantic vectors of candidate elements and test operation instructions.

[0038] Semantic vectors, in this context, refer to the numerical representations of text, instructions, or contextual information converted using natural language processing techniques. They transform unstructured semantic information into a computable mathematical form to facilitate similarity calculations and semantic alignment.

[0039] The semantic alignment score is a numerical value that quantifies the degree of semantic matching between the semantic vector of a candidate element and the semantic vector of a test operation instruction. A higher semantic alignment score indicates that the candidate element is more consistent with the test operation instruction in terms of function or intent.

[0040] Specifically, for candidate elements, their visible text content can be extracted as semantic information and converted into a model vector. For test operation instructions, such as "login button," the keywords "click" and "login button" can be extracted and converted into another model vector. Subsequently, the semantic alignment score is obtained by calculating the cosine similarity between the two model vectors.

[0041] Step S104: Locate the target element based on the matching score and semantic alignment score.

[0042] Specifically, after obtaining the matching score and semantic alignment score for each candidate element, a weighted average method can be used, such as directly adding the two scores to obtain a comprehensive score. Then, the candidate element with the highest comprehensive score is selected as the final target element for localization. If multiple candidate elements have the same comprehensive score, one can be selected randomly, or selected according to their order of appearance in the image.

[0043] It is understood that the script-adaptive element location method proposed in this application, by combining visual matching and semantic alignment, solves the problems of locator failure and misidentification of target elements in traditional UI automation testing when page structure changes. This method can get rid of the dependence on the original static attributes, improve the accuracy and robustness of location, reduce test script interruptions and a large amount of manual intervention caused by page changes, thereby achieving continuous and stable operation and efficiency improvement of automated testing.

[0044] In some of the embodiments described above in this application, a semantic alignment score is proposed to calculate the semantic alignment score between candidate elements and test operation instructions to assist in locating target elements. However, in its implementation, only a single candidate element semantic vector and operation semantic vector are simply constructed. The multi-dimensional semantic information of the candidate element itself and its surrounding environment is not fully explored, and the alignment calculation is not combined with multi-dimensional information. It is impossible to distinguish candidate elements with the same text content but different functions and located in different areas of the page. This can easily lead to inaccurate semantic alignment scores, which in turn can cause misidentification of target element location and fail to guarantee the accuracy of location.

[0045] In response, this application further proposes a script-adaptive element positioning method, such as... Figure 2 As shown, the method includes: Step S201: In response to the test operation command's failure to locate the target element in the current image, obtain the image information where the target element is located. The target element includes the element located by the attribute locator during the test. (See details...) Figure 1 Step S101 in the embodiment will not be repeated here.

[0046] Step S202: Obtain the candidate element list information from the image information, and calculate the matching score between the candidate elements in the candidate element list information and the corresponding elements in the historical image. The corresponding elements are those that are the same as the target element in the historical image. (See details...) Figure 1 Step S102 in the embodiment will not be described again here.

[0047] Step S203 involves constructing corresponding semantic vectors based on the candidate elements and test operation instructions, respectively, and calculating the semantic alignment score between the semantic vectors of the candidate elements and the semantic vectors of the test operation instructions. Specifically, this includes: Step S2031: Based on the candidate elements in the candidate element list, construct a text semantic vector and a context semantic vector representing the region near the candidate element. The region near the candidate element includes other elements.

[0048] Specifically, constructing text semantic vectors aims to capture the semantic information of the candidate element's own text content. For example, for a button, its text might be "Submit" or "Login," and the text semantic vector will encode the meaning of these words. One approach is to use a pre-trained word embedding model, such as Word2Vec, to convert each word in the candidate element's text content into a vector, and then aggregate these word vectors using averaging, weighted averaging, or sequence models such as LSTM or Transformer encoders to obtain a text semantic vector representing the entire text content. Another approach is to use a pre-trained language model based on the Transformer architecture, such as BERT, to encode the candidate element's text, extracting the word vectors representing the entire text from its output layer or performing pooling operations on all word vectors as the candidate element's text semantic vector.

[0049] Contextual semantic vectors, representing the region surrounding a candidate element, describe the environmental information of the candidate element within its page layout, i.e., the semantic background formed by other elements around it. This helps distinguish elements with the same text content but different functions or positions. One implementation involves first identifying other elements within a certain region around the candidate element, for example, through the DOM tree structure or visual layout, extracting information such as text, tags, and attributes of these elements, then serializing or constructing this information into a graph structure, and encoding it using a sequence encoder, such as LSTM, or a graph neural network, such as GCN, to generate a contextual semantic vector. Another implementation involves cropping an image patch containing the candidate element and its surrounding region from the current image, then using a pre-trained convolutional neural network, such as ResNet, to extract the visual features of this image patch, using these visual features as contextual semantic vectors, thus implicitly representing the visual context information surrounding the candidate element.

[0050] Step S2032: Construct an operation semantic vector based on the test operation instructions.

[0051] Specifically, constructing operational semantic vectors aims to represent the intent and semantic information contained in test operation instructions, such as operation types like "click," "input," and "swipe," as well as the operation objects, such as "login button" and "username input box." One approach is to use Natural Language Processing (NLP) models to parse and encode the text of the test operation instructions, for example, using sentence embedding models like Sentence-BERT to convert the entire instruction text into a vector that captures the overall semantics of the instruction. Another approach is to construct a rule-based or machine learning-based parser to extract key operational verbs and target nouns from the test operation instructions, and then map these extracted semantic units into a predefined concept vector space, or encode this structured information into operational semantic vectors using a specially trained neural network.

[0052] Step S2033: Calculate the semantic alignment score based on the text semantic vector, the context semantic vector, and the operation semantic vector.

[0053] Specifically, calculating the semantic alignment score is a quantitative metric used to measure the degree of matching between the textual semantics and contextual semantics of a candidate element and the semantic intent of the test operation instruction. One implementation involves concatenating the textual semantic vector and the contextual semantic vector of the candidate element, or integrating them through a fusion network, to form a comprehensive semantic representation of the candidate element. Then, the similarity between this comprehensive representation and the operation semantic vector is calculated, for example using cosine similarity, dot product, or prediction through a small neural network, to obtain the semantic alignment score. Another implementation involves designing a multi-head attention mechanism, allowing the operation semantic vector to interact separately with the textual semantic vector and the contextual semantic vector, calculating their respective alignment scores. These scores are then weighted and summed or fused through a gating mechanism to obtain the final semantic alignment score.

[0054] Step S204: Locate the target element based on the matching score and semantic alignment score. See details in [link / reference]. Figure 1 Step S104 in the embodiment will not be described again here.

[0055] It is understood that, through the above technical solution, this application, when calculating the semantic alignment score between candidate elements and test operation instructions, is no longer limited to a single dimension of semantic information, but comprehensively constructs text semantic vectors, context semantic vectors, and operation semantic vectors. Specifically, by constructing text semantic vectors, the textual meaning of the candidate element itself can be accurately captured, providing a foundation for semantic matching. Simultaneously, by constructing context semantic vectors representing the area near the candidate element, semantic information from other elements surrounding the candidate element is introduced, effectively compensating for the shortcomings of relying solely on its own text for judgment. This allows the system to distinguish candidate elements with the same text content but different page areas or functions. For example, it can distinguish between two elements on the page that both contain the word "login," one being the page title and the other a function button. Furthermore, by constructing operation semantic vectors, the true intent of the test operation instructions is integrated into the semantic matching process, ensuring that the semantic alignment score considers not only the semantics of the element itself but also the user's expected operation, thus making the matching results more consistent with the needs of the actual testing scenario. Ultimately, by calculating the semantic alignment score based on these three types of multi-dimensional semantic vectors, the semantic matching degree between candidate elements and test operation instructions can be reflected more comprehensively and accurately. This effectively reduces the risk of misidentification due to insufficient semantic understanding, significantly improves the accuracy and robustness of target element positioning, and thus ensures the stability of automated test scripts when the page structure changes.

[0056] In some of the solutions mentioned above in this application, a semantic alignment score is calculated based on text semantic vectors, context semantic vectors, and operation semantic vectors to determine the degree of matching between candidate elements and test operation instructions, thereby assisting in the localization of target elements. However, in this process, the matching degree of candidate elements is not evaluated from multiple different dimensions and then integrated. The score is obtained by relying solely on simple vector matching, which cannot distinguish between candidate elements with the same text content but different functions, positions, and contexts. This easily leads to inaccurate semantic alignment scores, resulting in misidentification of target element localization. This fails to solve the defect in the prior art where elements with the same text but different functions are easily confused in localization.

[0057] In this regard, this application further proposes the following steps for calculating the semantic alignment score based on the text semantic vector, context semantic vector, and operation semantic vector: Step a1: Calculate the text semantic similarity score of each candidate element by using the similarity between the text semantic vector and the operation semantic vector.

[0058] Step a2: Calculate the context matching score of each candidate element by the similarity between the context semantic vector and the element control semantic vector. The element control semantic vector is a pre-stored descriptive information used to characterize the shape of the target element.

[0059] Step a3: Determine the prior probability score of each candidate element based on the historical occurrence position of the target element and the current position of each candidate element.

[0060] Step a4: The text semantic similarity score, context matching score and position prior probability score of each candidate element are weighted and fused to obtain the semantic alignment score between each candidate element and the test operation command.

[0061] Specifically, a textual semantic similarity score is calculated for each candidate element by measuring the similarity between the textual semantic vector and the operational semantic vector. This aims to evaluate the degree of matching at the textual level between the textual content of the candidate element and the semantic intent expressed by the test operation instruction. In one implementation, a pre-trained word embedding model, such as Word2Vec, can be used to convert the textual information of the candidate element and the test operation instruction into high-dimensional vector representations, and then the textual semantic similarity is quantified by calculating the cosine similarity between these vectors. In another implementation, a deep learning-based semantic matching model can be used, for example, through a neural network structure to encode the candidate element text and the operation instruction separately, and calculate their similarity in a shared semantic space to obtain a textual semantic similarity score.

[0062] The context matching score for each candidate element is calculated by using the similarity between the context semantic vector and the element control semantic vector. The element control semantic vector is a pre-stored descriptive information representing the shape of the target element. It is used to evaluate the degree of matching between the candidate element's environmental information and its own shape features and the expected context and shape features of the target element. In one implementation, the context semantic vector can be generated by a feature extractor, such as a convolutional neural network or a graph neural network, from the DOM structure information surrounding the candidate element, the text content of adjacent elements, and visual layout features. The element control semantic vector can be a pre-stored vector describing typical visual features of the target element, such as button shape, input box style, or structural features, such as a specific combination of HTML tags. The context matching score is obtained by calculating the similarity between the vectors, such as the reciprocal of the Euclidean distance or cosine similarity. In another implementation, a classifier or regression model can be trained, taking the context semantic vector and the element control semantic vector as input, and directly outputting the context matching score. This model can learn the influence of different combinations of context and control features on the matching degree.

[0063] This method determines the prior location probability score for each candidate element based on its historical occurrence positions and the current positions of all candidate elements. The aim is to provide a location-based prior evaluation for the current candidate element by utilizing the location information of the target element during historical testing. In one implementation, the average coordinates of the target element or the region with the highest frequency of occurrence in historical images can be recorded. For each candidate element, the distance between its current position and the historical position of the target element is calculated (e.g., Euclidean distance or Manhattan distance), and this distance is converted into a prior location probability score using an inverse function; the closer the distance, the higher the score. In another implementation, the screen can be divided into several regions, and the historical frequency of the target element in each region can be statistically analyzed to form a location probability distribution map. When a candidate element appears in a certain region, its prior location probability score is the historical frequency of occurrence in that region.

[0064] The semantic similarity score, context matching score, and positional prior probability score of each candidate element are weighted and fused to obtain a semantic alignment score between each candidate element and the test operation command. The aim is to comprehensively consider matching information from different dimensions to generate a comprehensive and accurate semantic alignment score. One implementation method uses a linear weighted summation, assigning a weight coefficient to each score and then using a small neural network as a fusion module. Taking the three-dimensional scores as input, a non-linear fusion function is learned and outputs the final semantic alignment score, thus better capturing the complex relationships between different scores.

[0065] In one example, the semantic alignment score is obtained using the following formula: . in, For text semantic vectors, To manipulate semantic vectors, For contextual semantic vectors, For element control semantic vectors, Let be the prior probability of the location.

[0066] It is understood that, through the above technical solution, this application can evaluate the matching degree between candidate elements and target requirements from three different dimensions: text content, context, and location. Then, by integrating multi-dimensional scores, a final semantic alignment score is obtained, thereby effectively improving the accuracy of the semantic alignment score. Specifically, the text semantic similarity score can match the fit between candidate elements and test requirements at the text content level, clarifying the matching degree at the text content level; the context matching score combines the context information composed of other elements surrounding the candidate element and the shape features of the target element itself, which can distinguish candidate elements with the same text content but different environments and shapes, solving the problem of difficulty in distinguishing different functional elements of the same text; the location prior probability score uses historical information of the target element's location for prior evaluation, which can further narrow down the range of target candidate elements, improve the rationality of candidate element matching, and use historical information to assist in selecting the correct target element. By weightedly integrating the scores obtained from the three dimensions, multi-dimensional matching information is integrated, resulting in a comprehensive and accurate semantic alignment degree result for candidate elements. This provides an accurate scoring basis for target element positioning, avoids the misidentification problem caused by single-dimensional matching, and significantly improves the robustness and accuracy of script adaptive positioning.

[0067] In some embodiments described above, this application proposes obtaining a list of candidate elements from image information and calculating matching scores between candidate elements in the list and corresponding elements in historical images. This is used to adaptively locate target elements by combining semantic alignment scores after the original attribute locator fails to locate them. However, in its implementation, it does not distinguish whether the target element exists in the historical image. Regardless of whether the corresponding element exists in the historical image, the complete matching score calculation process is performed. This not only incurs unnecessary computational overhead and slows down the overall localization efficiency, but also generates incorrect matching scores when the corresponding element does not exist in the historical image, interfering with the subsequent localization judgment of target elements and leading to a decrease in localization accuracy.

[0068] In response, this application further proposes a script-adaptive element positioning method, such as... Figure 3 As shown, the method includes: Step S301: In response to the test operation command's failure to locate the target element in the current image, obtain the image information where the target element is located. The target element includes the element located by the attribute locator during the test. (See details...) Figure 1 Step S101 in the embodiment will not be repeated here.

[0069] Step S302: Obtain the candidate element list information from the image information, and calculate the matching score between the candidate elements in the candidate element list information and the corresponding elements in the historical image. The corresponding elements are those that are the same as the target element in the historical image. Specifically, this includes: Step S3021: Identify text regions in the image information and obtain candidate elements and coordinate information corresponding to each text region.

[0070] Step S3022: Sort each candidate element and its corresponding coordinate information to obtain a list of candidate elements.

[0071] Step S3023: Query historical images and determine whether the historical images contain the target local image of the corresponding element.

[0072] Step S3024: If it exists, obtain the feature information of the corresponding element in the target local image, calculate the matching degree between the feature information of the corresponding element and the feature information of the candidate element, and obtain the matching score value.

[0073] Step S3025: If it does not exist, then perform the steps of constructing corresponding semantic vectors based on candidate elements and test operation instructions respectively, and calculating the semantic alignment score between semantic vectors.

[0074] The process involves identifying text regions within an image to obtain candidate elements and their coordinates for each region. The aim is to initially filter potential target elements from the current test image by recognizing visual text content. Interactive elements or important information in a UI interface are often accompanied by text labels; therefore, identifying text regions is fundamental to efficiently and accurately acquiring candidate elements. Specifically, methods based on Optical Character Recognition (OCR) technology can be employed. For example, OCR technology can be used to detect and recognize text in the entire image, obtaining the bounding box coordinates and recognized text content for each identified text block, and then using these text blocks as candidate elements.

[0075] Each candidate element and its corresponding coordinate information are sorted to obtain a candidate element list. This aims to structure and organize the identified candidate elements and their coordinate information into an ordered list. An ordered candidate element list helps standardize subsequent matching, scoring, and localization processes, avoiding complex or erroneous processing logic caused by disorder. For example, candidate elements can be sorted in ascending order based on their Y-axis coordinates; if the Y-axis coordinates are similar, they can then be sorted in ascending order based on their X-axis coordinates. This method simulates the user's reading habits from top to bottom and left to right, helping to maintain visual consistency. Alternatively, candidate elements can be sorted in descending order based on their area in the image, or based on their depth in the DOM tree, prioritizing more prominent or deeper elements.

[0076] The goal of this approach is to efficiently determine whether image feature-based matching score calculations are needed by querying historical images and identifying whether the target element is present in a corresponding local image. By querying historical data, it's possible to determine if a visual record of the target element exists in historical images, thus avoiding unnecessary computation. Specifically, this can be achieved by maintaining a historical element database that stores an image snapshot and its feature descriptor for each target element at the time of successful localization. When a query is needed, the identifier of the current target element, such as text content or DOM path, is used to index and search the database to determine if a corresponding historical local image record exists. Another approach is to use image hashing or locality-sensitive hashing (LSH) techniques. Local image hash values ​​for target elements are pre-calculated and stored in historical images. During a query, the hash values ​​of potentially corresponding regions in the current image are calculated and compared with historical hash values ​​to determine if similar target local images exist.

[0077] If present, the feature information of the corresponding element in the target local image is obtained, and the matching degree between the feature information of the corresponding element and the feature information of the candidate element is calculated to obtain a matching score. When there is a visual record of the target element in the historical image, this step uses this historical information to compare the visual features with the currently identified candidate element to quantify the similarity between them, thereby obtaining a matching score. This score reflects the visual proximity of the candidate element to the historical target element. For example, local feature descriptors such as SIFT (Scale-Invariant Feature Transform) or ORB (Oriented FAST and Rotated BRIEF) can be used. First, feature points and descriptors are extracted from the historical target local image and the current candidate element image. Then, algorithms such as FLANN (Fast Library for Approximate Nearest Neighbors) or Brute-Force Matcher are used for feature matching, and the matching degree is calculated based on the number and quality of matching points. In addition, deep learning-based feature extractors, such as pre-trained convolutional neural network (CNN) models, such as ResNet and VGG, can be used as image features. The historical target local image and the current candidate element image are input into the CNN model, their feature vectors are extracted, and then the cosine similarity or Euclidean distance between the two feature vectors is calculated as the matching score.

[0078] If the target element does not exist, the system proceeds by constructing corresponding semantic vectors based on the candidate element and the test operation instruction, and calculating the semantic alignment score between these vectors. When no visual record of the target element is found in the historical images, this step skips the image feature-based matching score calculation and directly proceeds to the semantic alignment score process. This avoids invalid or potentially misleading image matching without historical visual references, thus improving efficiency and accuracy. This step means the system no longer relies on visual matching but instead utilizes text and contextual information for semantic-level matching. Specifically, it calls the subsequent semantic processing module, which constructs semantic vectors using Natural Language Processing (NLP) techniques based on the text content of the candidate element, contextual information, and the text content of the test operation instruction, and calculates the semantic alignment score between these vectors.

[0079] Step S303 involves constructing corresponding semantic vectors based on the candidate elements and test operation instructions, respectively, and calculating the semantic alignment score between the semantic vectors of the candidate elements and the semantic vectors of the test operation instructions. Specifically, this includes: Step S3031: Based on the candidate elements in the candidate element list, construct a text semantic vector and a contextual semantic vector representing the region near the candidate element. The region near the candidate element includes other elements. (See details...) Figure 2 Step S2031 in the embodiment will not be described again here.

[0080] Step S3032: Construct an operation semantic vector based on the test operation instructions. See details in [link / reference]. Figure 2 Step S2032 in the embodiment will not be described again here.

[0081] Step S3033: Calculate the semantic alignment score based on the text semantic vector, context semantic vector, and operation semantic vector. See details... Figure 2 Step S2033 in the embodiment will not be repeated here.

[0082] Step S304: Locate the target element based on the matching score and semantic alignment score. See details in [link / reference]. Figure 1 Step S104 in the embodiment will not be described again here.

[0083] It is understood that, through the above technical solution, this application introduces a pre-judgment mechanism to determine whether a target local image of the corresponding element exists in historical images during the process of obtaining the candidate element list information in image information and calculating the matching score. Specifically, candidate elements are efficiently and accurately filtered out by identifying text regions in the image information and then organized in an orderly manner. Subsequently, before calculating the matching score, the system first queries historical images to determine whether a target local image exists. If a corresponding target local image exists in the historical image, its feature information is matched with the feature information of the current candidate element to obtain a reliable matching score. Conversely, if a corresponding target local image does not exist in the historical image, the system intelligently skips the image feature matching step and directly proceeds to the step of constructing a semantic vector based on the candidate element and test operation instructions and calculating the semantic alignment score. This effectively avoids unnecessary image feature matching calculations when there is no historical visual reference, significantly saves computing resources, and improves the overall positioning speed. At the same time, it also avoids generating incorrect matching scores when historical records are missing, thereby eliminating interference with subsequent target element positioning judgments and effectively ensuring the accuracy of positioning.

[0084] In some of the solutions mentioned above in this application, a combination of matching score and semantic alignment score is proposed to locate target elements in order to achieve adaptive element location. However, in this process, the original method of fusing the two scores mostly uses fixed weights, which cannot dynamically adjust the weight ratio of the two scores according to the runtime test environment. The reference value of the two scores is not the same in different test environments. For example, when the DOM structure changes significantly, the reference value of the matching score obtained based on historical images will decrease. When the network latency is high, the semantic calculation process is easily affected, and the matching score based on history is more reliable. Fixed weights cannot adapt to these changes in different scenarios, which can easily lead to location errors, increase the probability of secondary location failure, and fail to guarantee the accuracy and stability of element location in different test environments.

[0085] In response, this application further proposes a script-adaptive element positioning method, such as... Figure 4 As shown, the method includes: Step S401: In response to the test operation command's failure to locate the target element in the current image, obtain the image information where the target element is located. The target element includes the element located by the attribute locator during the test. (See details...) Figure 1 Step S101 in the embodiment will not be repeated here.

[0086] Step S402: Obtain the candidate element list information from the image information, and calculate the matching score between the candidate elements in the candidate element list information and the corresponding elements in the historical image. The corresponding elements are those that are the same as the target element in the historical image. Specifically, this includes: Step S4021: Identify text regions in the image information and obtain candidate elements and coordinate information corresponding to each text region. See details... Figure 3 Step S3021 in the embodiment will not be described again here.

[0087] Step S4022: Sort each candidate element and its corresponding coordinate information to obtain a candidate element list. See details... Figure 3 Step S3022 in the embodiment will not be described again here.

[0088] Step S4023: Query historical images and determine whether the historical images contain the target local image with the corresponding element. See details. Figure 3 Step S3023 in the embodiment will not be repeated here.

[0089] Step S4024: If it exists, obtain the feature information of the corresponding element in the target local image, and calculate the matching degree between the feature information of the corresponding element and the feature information of the candidate element to obtain the matching score. See details. Figure 3 Step S3024 in the embodiment will not be described again here.

[0090] Step S4025: If the candidate element does not exist, proceed to construct corresponding semantic vectors based on the candidate element and the test operation instruction, and calculate the semantic alignment score between the semantic vectors. See details. Figure 3 Step S3025 in the embodiment will not be described again here.

[0091] Step S403 involves constructing corresponding semantic vectors based on the candidate elements and test operation instructions, respectively, and calculating the semantic alignment score between the semantic vectors of the candidate elements and the semantic vectors of the test operation instructions. Specifically, this includes: Step S4031: Based on the candidate elements in the candidate element list, construct a text semantic vector and a contextual semantic vector representing the region near the candidate element. The region near the candidate element includes other elements. (See details...) Figure 2 Step S2031 in the embodiment will not be described again here.

[0092] Step S4032: Construct an operation semantic vector based on the test operation instructions. See details in [link / reference]. Figure 2 Step S2032 in the embodiment will not be described again here.

[0093] Step S4033: Calculate the semantic alignment score based on the text semantic vector, context semantic vector, and operation semantic vector. See details... Figure 2 Step S2033 in the embodiment will not be repeated here.

[0094] Step S404: Locate the target element based on the matching score and semantic alignment score. Specifically, this includes: Step S4041: Determine the test environment status, which includes at least the DOM structure change status and network latency status.

[0095] Step S4042: Determine the weight coefficients of the matching score and semantic alignment score based on the test environment status.

[0096] Step S4043: Multiply the matching score and semantic alignment score by their respective weight coefficients and sum them to obtain the comprehensive score; determine the target element based on the comprehensive score.

[0097] Specifically, determining the test environment state refers to perceiving and evaluating external or internal conditions that affect the accuracy and stability of element location during automated testing. DOM structure change state refers to the degree of change in the DOM structure of the application under test's user interface (UI) document object model, such as changes in element IDs, XPath paths, class names, or changes in the relative position or hierarchy of elements on the page. Network latency state refers to the latency of data transmission between the test environment and the application under test's server, typically manifested as slow page loading speeds and long element rendering times. Determining the test environment state can be achieved in various ways. For example, DOM structure changes can be detected by continuously monitoring the hash value or checksum of the DOM tree; when the hash value or checksum changes, the DOM structure is considered to have changed. Alternatively, the amount of change can be calculated by comparing the current DOM structure with a baseline DOM structure. For network latency state, it can be obtained by sending network requests to a specific server and measuring the response time, such as ping values ​​or HTTP request response times, or by monitoring the network request waterfall chart in the browser's developer tools to evaluate page loading performance.

[0098] After determining the test environment state, weight coefficients for the matching score and semantic alignment score are determined based on this state. These weight coefficients are numerical factors used to adjust the contribution of the matching score and semantic alignment score to the final overall score. The weight coefficients are not fixed but dynamically adjusted based on the currently detected test environment state. Methods for determining the weight coefficients can include, but are not limited to, predefined rule base matching and machine learning model prediction. In the predefined rule base matching method, a set of weight adjustment rules can be predefined based on different test environment states, such as large DOM structure changes, high network latency, stable DOM structure, and low network latency. For example, when the DOM structure change state indicates large changes, the weight coefficient corresponding to the matching score can be decreased, while the weight coefficient corresponding to the semantic alignment score can be increased; when the network latency state indicates high latency, the weight coefficient corresponding to the matching score can be increased, while the weight coefficient corresponding to the semantic alignment score can be decreased. In the machine learning model prediction method, a machine learning model can be trained using historical test data. This model can dynamically predict the most suitable weight coefficients for the matching score and semantic alignment score based on the current test environment state to maximize the localization success rate.

[0099] Subsequently, the matching score and semantic alignment score are multiplied by their respective weighting coefficients and then summed to obtain the comprehensive score. The comprehensive score is the final evaluation value that combines visual matching information and semantic understanding information, and dynamically adjusts their relative importance according to the current testing environment. This step obtains a single, comprehensive evaluation index that fully reflects the degree of conformity between the candidate element and the target element by multiplying the previously calculated matching score and semantic alignment score by their corresponding weighting coefficients and then summing the two weighted scores.

[0100] In one example, the test environment state is represented by the environment perception factor E, and the weighting coefficient of the matching score is... The weighting coefficient for the semantic alignment score is .

[0101] in, and It's a function that adjusts based on environmental conditions, such as DOM structure stability, page load time, and network latency. When the DOM changes frequently, it reduces the weight of traditional positioning and increases the weight of OCR.

[0102] When network latency is high, pre-loaded visual features are prioritized for localization. The final decision is made using weighted fusion to obtain a comprehensive score. . To match the score value, This is the semantic alignment score.

[0103] Finally, the target element is determined based on the overall score. Determining the target element means selecting the element that best meets the test operation instructions from the candidate element list based on the calculated overall score. This step can be implemented by selecting the candidate element with the highest overall score, or selecting a candidate element whose overall score exceeds a preset threshold. Typically, the system iterates through all candidate elements, calculates the overall score for each, and then selects the candidate element with the highest overall score as the final target element. In some scenarios, an overall score threshold can be set; all candidate elements with scores exceeding this threshold are considered potential target elements. Further filtering can then be performed based on other auxiliary information, such as the element's visibility and interactivity on the page, or the first element that meets the criteria can be selected from multiple high-scoring elements.

[0104] It is understood that, through the above technical solution, this application can effectively solve the problem of the fixed weight fusion strategy failing due to changes in the test environment state during element localization. Upon responding to a test operation command's failure to locate a target element in the current image, the system first obtains the image information where the target element is located and identifies a list of candidate elements. Simultaneously, it calculates the matching score between the candidate element and the corresponding element in the historical image, as well as the semantic alignment score between the candidate element and the test operation command. Based on this, this application further introduces dynamic awareness of the test environment state. By determining the current DOM structure change state and network latency state, it can evaluate the reliability of the matching score and semantic alignment score in the current environment in real time. For example, when the DOM structure changes significantly, the traditional visual matching-based matching score may no longer be reliable. In this case, the system will reduce its weight and rely more on the semantic alignment score. Conversely, when network latency is high, which may affect the accuracy of semantic model inference, the system will increase the weight of the matching score to ensure the stability of localization. This allows the comprehensive score to more accurately reflect the true conformity of the candidate element in the current environment, avoiding the limitations of the fixed weight strategy in different scenarios. Ultimately, the target element is determined based on the comprehensive score after dynamic weighted fusion, which significantly improves the accuracy and stability of element localization in complex and variable testing environments, reduces the probability of secondary localization failure, and thus achieves more intelligent and robust script-adaptive element localization.

[0105] In some embodiments of this application, this application further proposes to determine the weight coefficients of the matching score and the semantic alignment score based on the test environment state, specifically including: when the amount of change representing the change state of the DOM structure is greater than a preset change threshold, the weight coefficient corresponding to the matching score is reduced and the weight coefficient corresponding to the semantic alignment score is increased; when the network latency is greater than a preset latency threshold, the weight coefficient corresponding to the matching score is increased and the weight coefficient corresponding to the semantic alignment score is reduced.

[0106] The change in DOM structure is used to quantify the difference between the current page's Document Object Model (DOM) structure and the historical DOM structure. It can be obtained by calculating the edit distance between the current DOM tree and the historical DOM tree, or by comparing the number of changes in the XPath paths or CSS selectors of key elements. The preset change threshold is a pre-defined value used to determine whether the DOM structure change warrants an adjustment to the positioning strategy. When the actual measured DOM structure change exceeds this threshold, it indicates a significant change in the page structure. This threshold can be obtained through statistical analysis of historical test data, for example, set as a percentile of the average DOM change, or empirically set as a specific number of node changes. Reducing the weight coefficient corresponding to the matching score means reducing the influence of the matching score in the overall score calculation. This can be achieved by multiplying the original weight coefficient by a factor less than 1, or by selecting a lower weight value from the preset weight adjustment table. Increasing the weight coefficient corresponding to the semantic alignment score means increasing the influence of the semantic alignment score in the overall score calculation. This can be achieved by multiplying the original weight coefficient by a factor greater than 1, or by selecting a higher weight value from the preset weight adjustment table.

[0107] In addition, network latency status reflects the network connection quality and page loading speed of the current test environment. Its purpose is to identify situations where page elements may not be fully loaded or rendered. For example, this can be obtained by monitoring page loading completion time, the loading time of specific key resources such as main images and script files, or by performing network latency tests on the target server, such as ping values. The preset latency threshold is a pre-set time value used to determine whether network latency has reached a level requiring adjustment of the positioning strategy. When the actual measured network latency exceeds this threshold, it indicates a poor network environment, and the page may not be fully stable. This threshold can be set according to user experience standards or system performance requirements, or dynamically adjusted based on the average latency of historical test environments. Increasing the weight coefficient corresponding to the matching score means increasing the influence of the matching score in the overall score calculation. This can be achieved by multiplying the original weight coefficient by a factor greater than 1, or by selecting a higher weight value from the preset weight adjustment table. Decreasing the weight coefficient corresponding to the semantic alignment score means reducing the influence of the semantic alignment score in the overall score calculation. This can be achieved by multiplying the original weight coefficient by a factor less than 1, or by selecting a lower weight value from the preset weight adjustment table.

[0108] It is understood that, through the above technical solution, this application can dynamically adjust the weight coefficients of the matching score and semantic alignment score according to different test environment states, thereby overcoming the limitation of traditional fixed weight strategies that cannot adapt to changing test environments. The weight adjustment mechanism allows the comprehensive score to more accurately reflect the probability that the current candidate element is the target element, significantly improving the accuracy and success rate of target element localization in various complex and dynamic test environments, effectively avoiding secondary localization failures caused by environmental changes, and thus improving the robustness and stability of UI automated testing.

[0109] In some embodiments of this application, the above-mentioned method further includes: Step b1: Save the process information of successfully locating the target element and the corresponding image information.

[0110] Step b2: Based on the process information and image information, generate a positioning strategy for the target element.

[0111] Step b3: When locating the target element again, use the location strategy to locate the target element.

[0112] Specifically, "saving process information and corresponding image information for successfully locating a target element" aims to record and store all key data involved in a successful element relocation process. Process information may include, but is not limited to: original locator information that caused the initial location failure, test operation instructions that triggered relocation, matching score and semantic alignment score calculated during relocation, the comprehensive score used to finally determine the target element, and the test environment status evaluated during the location decision-making process, such as DOM structure changes and network latency. This process information can be stored in the form of structured logs, database records, or specific data files. Image information refers to a visual snapshot of the user interface (UI) when the target element is successfully located; it can be a complete page screenshot or a partial image containing only the target element and its surrounding area. This image information can be stored in a standard image format and associated with the corresponding process information. By saving this detailed information, this application provides a comprehensive and realistic data foundation for subsequent location strategy generation, ensuring the integrity and effectiveness of the accumulated experience.

[0113] The goal of "generating a location strategy for target elements based on process and image information" is to extract reusable rules or models from saved successful location experiences. The methods for generating location strategies can be diverse. For example, based on the analysis of process information, rules can be extracted to determine which location method is more effective under specific environmental conditions, such as significant changes in the DOM structure, or which weight combinations are optimal. These rules can be encoded as conditional statements or decision trees. Another approach is to use process information as features and the final successful location path or method as labels to train a lightweight machine learning model, such as a decision tree model, support vector machine, or small neural network, which can predict the optimal location scheme based on new inputs. Furthermore, a more robust new locator can be automatically generated based on the results of successful location, such as a more stable XPath expression, CSS selector, or visual feature-based locator, to replace the original, easily failing locator. In this way, this application transforms one-time successful location experiences into reusable intelligent strategies, providing guidance for subsequent automated location.

[0114] The aim of "using the positioning strategy to locate the target element when relocating it" is to apply the generated positioning strategy to real-world test scenarios to improve positioning efficiency and success rate. When the system attempts to locate the same target element again, it first checks if a learned positioning strategy exists for that target element. If it does, the system will prioritize using that strategy. For example, if the strategy indicates that semantic alignment should be prioritized and given higher weight in the current DOM structure change state, the system will directly execute this strategy without having to re-perform a comprehensive dynamic weight determination process for matching scores and semantic alignment scores. If the strategy is a new, more robust locator, the system will directly attempt to use that new locator for positioning. By directly applying the learned strategy, this application avoids repeating the entire repositioning calculation process after each positioning failure, thereby significantly reducing computational resource consumption and positioning time.

[0115] It is understood that, through the above technical solution, this application solves the problem of the original solution requiring repeated full-process calculations every time the target element is relocated by accumulating and reusing experience in successfully locating the target element. This achieves continuous optimization of element location capabilities and forms an automated location closed loop. When locating the target element again, the generated location strategy is used, eliminating the need to re-execute the full matching calculation after a location failure, reducing unnecessary repetitive calculations, and allowing the successful location experience to be effectively reused. This gradually optimizes the overall location capability, forming an automatic location closed loop that does not require repeated manual intervention, achieving continuous evolution of location capabilities, and thus effectively improving the efficiency and stability of repeated location.

[0116] In one example, the following provides a more specific illustration of the above technical solution, such as... Figure 5 , Figure 6 As shown: Suppose an automated test script is designed to click a "Login" button on a test webpage. Due to a recent UI redesign of the website, the XPath or ID attributes previously used to locate the "Login" button are no longer valid. As a result, the test script cannot find the target element when executing this step, thus triggering a location failure event.

[0117] When the test command "Click the login button" is executed, the system attempts to locate the "Login" button on the current page using a preset attribute locator. Due to a UI redesign, this locator becomes ineffective, and the system responds to this location failure event. At this point, the system immediately captures a complete screenshot of the current page as image information of the target element. This image information includes all visible elements on the page and their layout.

[0118] After receiving image information, the system analyzes the image using a first-channel model, such as a computer vision-based model. This model first identifies all text regions in the image; for example, it might identify text like "login," "execute," or "details." Each identified text region and its corresponding coordinates are considered a candidate element. These candidate elements and their corresponding coordinates are then sorted to form a candidate element list.

[0119] Next, the system queries the historical image database to find historical images where the "Login" button was successfully located. If a target local image containing the corresponding element of the "Login" button is found, the system extracts the visual feature information of the corresponding element from that historical local image, such as the button's shape, color, size, and icon. Then, the system calculates the matching degree between this historical feature information and the visual feature information of each candidate element in the current image. For example, it compares the visual features of the historical "Login" button with the visual features of a candidate button named "Login" in the current image. The calculation result yields a matching score, which quantifies the visual similarity between the current candidate element and the historical target element. This step, through visual perception, can provide preliminary location clues based on appearance similarity even if the underlying attributes of the element change.

[0120] Simultaneously, the system uses a second-channel model, such as a natural language processing-based model, to perform semantic analysis on candidate elements and test operation instructions. For each candidate element in the candidate element list, such as the "login" button candidate in the current image, the system constructs: Text semantic vector: based on the text content of the candidate element itself, such as "login".

[0121] Contextual semantic vector: Text and layout information based on other elements in the vicinity of the candidate element, such as "Execute" and "Details".

[0122] For the test operation command "Click the login button", the system constructs an operation semantic vector.

[0123] The system then calculates the semantic alignment score based on these semantic vectors: Textual semantic similarity score: Calculates the similarity between the textual semantic vector of a candidate element and the operational semantic vector. This helps determine the degree of match between the textual content of the candidate element and the operational intent.

[0124] Contextual Matching Score: This score calculates the similarity between the contextual semantic vector of a candidate element and the pre-stored semantic vectors of element controls. Element control semantic vectors are descriptive information used to represent target elements, such as buttons, typical shapes, and functional characteristics. This helps determine whether the context in which the candidate element exists matches the expectations of the target element.

[0125] Prior location probability score: Based on the location of the target element in the historical image and the current location of each candidate element, determine the prior location probability score of each candidate element.

[0126] Finally, the system performs a weighted fusion of the textual semantic similarity score, context matching score, and positional prior probability score for each candidate element to obtain a semantic alignment score between each candidate element and the test operation instruction. This step, through a deep understanding of the text, context, and operation intent, effectively distinguishes elements that are only similar in text but different in function or semantics, such as "Login" in the page title and the actual "Login" button, thus solving the problem of insufficient semantic understanding in existing technologies.

[0127] After obtaining the matching score and semantic alignment score of each candidate element, the system needs to make a comprehensive judgment to determine the final target element.

[0128] First, the system determines the current state of the test environment, which includes at least the DOM structure change status, such as whether the page DOM structure has undergone large-scale changes, and the network latency status, such as the current page loading speed.

[0129] Based on these test environment conditions, the system dynamically determines the weighting coefficients for the matching score and semantic alignment score. For example: When the amount of change representing the DOM structure changes exceeds the preset change threshold, it indicates that the page structure has changed significantly. The system will reduce the weight coefficient corresponding to the matching score because visual similarity may be unreliable due to reconstruction, and increase the weight coefficient corresponding to the semantic alignment score because semantic intent may be relatively stable.

[0130] When the network latency exceeds the preset latency threshold, it indicates that the page is loading slowly. The system will increase the weight coefficient corresponding to the matching score because visual features may be more stable when not fully loaded, and decrease the weight coefficient corresponding to the semantic alignment score.

[0131] The system multiplies the matching score and semantic alignment score by their respective weight coefficients and then sums them to obtain the comprehensive score for each candidate element.

[0132] Ultimately, based on the comprehensive score, the system selects the candidate element with the highest score as the target element, the "Login" button. This overcomes the rigidity of the decision-making mechanism in existing technologies, enabling the positioning strategy to adaptively adjust according to the runtime environment, thus improving the accuracy and robustness of positioning.

[0133] Once the "Login" button is successfully located, the system saves information about the successful location process, including the score used, weights, the final selected candidate elements, and the corresponding image information. Based on this information, the system generates a location strategy for the "Login" button. This strategy may be a more robust locator or a set of location rules combining visual and semantic features. When the test script needs to locate the "Login" button again, the system will prioritize using this newly generated location strategy. This gives the system the ability to learn and evolve, avoiding the need for repeated repairs after each failure, achieving a closed loop and continuous optimization in automated testing, and solving the problem of lacking evolutionary capabilities in existing technologies.

[0134] This embodiment also provides a script-adaptive element positioning device for implementing the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0135] This embodiment provides a script-adaptive element positioning device, such as... Figure 7 As shown, it includes: The acquisition module 701 is used to acquire image information of the target element in response to a test operation command for a failure to locate the target element in the current image. The target element includes the element located by the attribute locator during the test.

[0136] The first calculation module 702 is used to obtain candidate element list information in the image information and calculate the matching score between the candidate element in the candidate element list information and the corresponding element in the historical image, wherein the corresponding element is the same as the target element in the historical image.

[0137] The second calculation module 703 is used to construct corresponding semantic vectors based on candidate elements and test operation instructions respectively, and to calculate the semantic alignment score between the semantic vectors of candidate elements and the semantic vectors of test operation instructions.

[0138] The localization module 704 is used to locate the target element based on the matching score and the semantic alignment score.

[0139] In some alternative implementations, the second computing module 703 includes: The first unit is used to construct a text semantic vector and a context semantic vector representing the region near the candidate element based on the candidate element in the candidate element list. The region near the candidate element includes other elements. The second unit is used to construct operation semantic vectors based on test operation instructions; The third unit is used to calculate the semantic alignment score based on the text semantic vector, the context semantic vector, and the operation semantic vector.

[0140] The script-adaptive element positioning device provided in this application can execute the script-adaptive element positioning method provided in any embodiment of this application, and has the corresponding functional modules and beneficial effects of the method execution. Further functional descriptions of the above modules and units are the same as those in the corresponding embodiments described above, and will not be repeated here.

[0141] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0142] The following is a detailed reference. Figure 8 The diagram illustrates a structural schematic suitable for implementing the electronic device described in the embodiments of this application. The electronic device may include a processor (e.g., a central processing unit, graphics processor, etc.) 801, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 802 or a program loaded from memory 808 into random access memory (RAM) 803. The RAM 803 also stores various programs and data required for the operation of the electronic device. The processor 801, ROM 802, and RAM 803 are interconnected via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0143] Typically, the following devices can be connected to I / O interface 805: input devices 806 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 807 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; memory devices 808 including, for example, magnetic tapes, hard disks, etc.; and communication devices 809. Communication device 809 allows electronic devices to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 8 Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown, and more or fewer devices may be implemented or have instead.

[0144] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 809, or installed from a memory 808, or installed from a ROM 802. When the computer program is executed by the processor 801, it performs the functions defined in the script-adaptive element location method of embodiments of this application.

[0145] Figure 8 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0146] This application also provides a computer-readable storage medium. The methods described in this application can be implemented in hardware or firmware, or implemented as recordable on a storage medium, or implemented as computer code downloaded over a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and then stored on a local storage medium. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium can also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code. When the software or computer code is accessed and executed by the computer, processor, or hardware, the script-adaptive element location method shown in the above embodiments is implemented.

[0147] A portion of this application can be applied as a computer program product, such as computer program instructions, which, when executed by a computer, can invoke or provide the methods and / or technical solutions according to this application through the operation of the computer. Those skilled in the art will understand that the forms in which computer program instructions exist in a computer-readable medium include, but are not limited to, source files, executable files, installation package files, etc. Correspondingly, the ways in which computer program instructions are executed by a computer include, but are not limited to: the computer directly executing the instructions, or the computer compiling the instructions and then executing the corresponding compiled program, or the computer reading and executing the instructions, or the computer reading and installing the instructions and then executing the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to a computer.

[0148] Although embodiments of this application have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of this application, and all such modifications and variations fall within the scope defined by the appended claims.

Claims

1. A script-adaptive element positioning method, characterized in that, The method includes: In response to a test operation command's failure to locate a target element in the current image, image information of the target element is obtained, wherein the target element includes the element located by the attribute locator during the test. Obtain candidate element list information from the image information, and calculate the matching score between the candidate elements in the candidate element list information and the corresponding elements in the historical image, wherein the corresponding elements are the same as the target element in the historical image; Construct corresponding semantic vectors based on the candidate elements and the test operation instructions respectively, and calculate the semantic alignment score between the semantic vectors of the candidate elements and the semantic vectors of the test operation instructions; The target element is located based on the matching score and the semantic alignment score.

2. The method according to claim 1, characterized in that, The step of constructing corresponding semantic vectors based on the candidate elements and the test operation instructions, and calculating the semantic alignment score between the semantic vectors of the candidate elements and the semantic vectors of the test operation instructions, includes: Based on the candidate elements in the candidate element list, construct a text semantic vector and a context semantic vector representing the region near the candidate element, wherein the region near the candidate element includes other elements; Construct an operation semantic vector based on the test operation instructions; The semantic alignment score is calculated based on the text semantic vector, the context semantic vector, and the operation semantic vector.

3. The method according to claim 2, characterized in that, The step of calculating the semantic alignment score based on the text semantic vector, the context semantic vector, and the operation semantic vector includes: The text semantic similarity score of each candidate element is calculated by the similarity between the text semantic vector and the operation semantic vector. The context matching score of each candidate element is calculated by the similarity between the context semantic vector and the element control semantic vector. The element control semantic vector is a pre-stored descriptive information used to characterize the shape of the target element. The prior probability score of each candidate element is determined based on the historical occurrence position of the target element and the current position of each candidate element. The semantic similarity score, context matching score, and positional prior probability score of each candidate element are weighted and fused to obtain the semantic alignment score between each candidate element and the test operation instruction.

4. The method according to claim 1, characterized in that, The step of obtaining the candidate element list information in the image information and calculating the matching score value between the candidate elements in the candidate element list information and the corresponding elements in the historical image includes: Identify text regions in the image information to obtain candidate elements and coordinate information corresponding to each text region; Sort each candidate element and its corresponding coordinate information to obtain the candidate element list information; Query historical images and determine whether the historical images contain the target local image of the corresponding element; If it exists, the feature information of the corresponding element in the target local image is obtained, and the matching degree between the feature information of the corresponding element and the feature information of the candidate element is calculated to obtain the matching score value. If none exists, then the steps of constructing corresponding semantic vectors based on the candidate elements and the test operation instructions respectively, and calculating the semantic alignment score between the semantic vectors are performed.

5. The method according to claim 1, characterized in that, The step of locating the target element based on the matching score and the semantic alignment score includes: Determine the test environment status, which includes at least the DOM structure change status and network latency status; Based on the test environment state, determine the weight coefficients of the matching score and the semantic alignment score, respectively; The matching score and the semantic alignment score are multiplied by their respective weight coefficients and then summed to obtain the comprehensive score. The target element is determined based on the comprehensive score.

6. The method according to claim 5, characterized in that, The step of determining the weight coefficients of the matching score and the semantic alignment score based on the test environment state includes: When the amount of change representing the change state of the DOM structure is greater than a preset change threshold, the weight coefficient corresponding to the matching score value is reduced, and the weight coefficient corresponding to the semantic alignment score value is increased. When the network latency is greater than a preset latency threshold, the weight coefficient corresponding to the matching score is increased, and the weight coefficient corresponding to the semantic alignment score is decreased.

7. The method according to any one of claims 1-6, characterized in that, The method further includes: Save the process information and corresponding image information of successfully locating the target element; Based on the process information and the image information, a positioning strategy for locating target elements is generated; When the target element is located again, the location strategy is used to locate the target element.

8. A script-adaptive element positioning device, characterized in that, The device includes: The acquisition module is used to acquire image information where the target element is located in response to a test operation command for a failure to locate a target element in the current image. The target element includes the element located by the attribute locator during the test. The first calculation module is used to obtain candidate element list information in the image information and calculate the matching score between the candidate elements in the candidate element list information and the corresponding elements in the historical image, wherein the corresponding elements are the same as the target element in the historical image. The second calculation module is used to construct corresponding semantic vectors based on the candidate elements and the test operation instructions respectively, and to calculate the semantic alignment score between the semantic vectors of the candidate elements and the semantic vectors of the test operation instructions. The localization module is used to locate the target element based on the matching score and the semantic alignment score.

9. An electronic device, characterized in that, include: A memory and a processor are communicatively connected, the memory storing computer instructions, and the processor executing the computer instructions to perform the script-adaptive element location method of any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing the computer to execute the script-adaptive element location method according to any one of claims 1 to 7.

Images