Element verification method and device, storage medium and electronic equipment

Abstract

Embodiments of the present application provide an element verification method and device, a storage medium and an electronic device, wherein the method comprises: determining a target element to be tested and function semantic information of the target element according to a test instruction of a target object; detecting an interface image of a target interface to obtain an element set on the interface image and determine a feature map of each element in the element set; determining a target feature map corresponding to the target element according to the function semantic information and the feature map of each element, and determining position information of the target element according to the target feature map; determining a test operation according to the test instruction and the position information, and executing the test operation to verify the target element.

Description

Technical Field

[0001] This application relates to the field of communications, and more specifically, to a method and apparatus for verifying elements, a storage medium, and an electronic device. Background Technology

[0002] In existing user interface (UI) automated testing technologies, the fixed attributes of UI elements, such as ID, XPath, and pixel coordinates, which are highly dependent on manual maintenance, immediately become invalid when the application interface undergoes version iteration, layout adjustment, or visual style change. Testers need to manually check and modify scripts version by version, resulting in low testing efficiency.

[0003] There is currently no effective solution to the problem of low testing efficiency caused by the need for frequent manual maintenance of fixed element attributes in related technologies during interface version iterations or layout adjustments.

[0004] Therefore, it is necessary to improve the relevant technology to overcome the aforementioned defects. Summary of the Invention

[0005] This application provides a method, apparatus, storage medium, and electronic device for verifying elements, in order to at least solve the problem in the related art where fixed attributes of elements that rely on manual maintenance require frequent manual maintenance of scripts during interface version iterations or layout adjustments, resulting in low testing efficiency.

[0006] According to one embodiment of this application, an element verification method is provided, comprising: determining a target element to be tested and the functional semantic information of the target element according to a test instruction of a target object; detecting an interface image of a target interface to obtain a set of elements on the interface image, and determining a feature map of each element in the set of elements; determining a target feature map corresponding to the target element according to the functional semantic information and the feature map of each element, and determining the position information of the target element according to the target feature map; determining a test operation according to the test instruction and the position information, and executing the test operation to verify the target element.

[0007] According to another embodiment of this application, an element verification device is provided, comprising: a first determining module, configured to determine a target element to be tested and the functional semantic information of the target element according to a test instruction of a target object; a second determining module, configured to detect an interface image of a target interface to obtain a set of elements on the interface image, and determine a feature map of each element in the set of elements; a third determining module, configured to determine a target feature map corresponding to the target element according to the functional semantic information and the feature map of each element, and determine the position information of the target element according to the target feature map; and a verification module, configured to determine a test operation according to the test instruction and the position information, and execute the test operation to verify the target element.

[0008] According to yet another embodiment of this application, a computer-readable storage medium is also provided, wherein a computer program is stored therein, and the computer program is configured to perform the steps in any of the above method embodiments when it is run.

[0009] According to yet another embodiment of this application, an electronic device is also provided, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0010] According to yet another embodiment of this application, a computer program product is also provided, including a computer program that, when executed by a processor, implements the steps in any of the above method embodiments.

[0011] This application parses the test instructions input by the target object to determine the functional semantic information of the target element. Subsequently, the interface image of the target interface is parsed to generate a feature map of each element in the element set on the target interface. During the matching phase, the functional semantic information is dynamically matched with each feature map in the element set to filter out the target element with the best semantic match, and the pixel coordinates of the target element are output based on its target feature map. This application does not rely on a preset element coordinate mapping table. Even if the interface layout is reconstructed, element positions are shifted, text content is changed, or visual styles are updated, the target element can still be located through the consistency between the feature map and the functional semantic information. This solves the problem in related technologies where manually maintained fixed element attributes require frequent script maintenance during interface version iterations or layout adjustments, leading to low testing efficiency. Attached Figure Description

[0012] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0013] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, those skilled in the art can obtain other drawings based on these drawings without creative effort.

[0014] Figure 1 This is a hardware structure block diagram of a computer device for an element verification method according to an embodiment of this application;

[0015] Figure 2 This is a flowchart (a) of an element verification method according to an embodiment of this application;

[0016] Figure 3 This is a schematic diagram of the feature map according to an embodiment of this application;

[0017] Figure 4 This is a schematic diagram of a target interface acquisition method according to an embodiment of this application;

[0018] Figure 5 This is a flowchart of a knowledge base update method according to an embodiment of this application;

[0019] Figure 6 This is a schematic diagram of an element verification system according to an embodiment of this application;

[0020] Figure 7 This is a flowchart (II) of an element verification method according to an embodiment of this application.

[0021] Figure 8 This is a structural block diagram of an element verification device according to an embodiment of this application. Detailed Implementation

[0022] The embodiments of this application will be described in detail below with reference to the accompanying drawings and examples.

[0023] It should be noted that the terms "first," "second," etc., in the specification, claims, and drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0024] The methods and embodiments provided in this application can be executed in a computer device or similar computing device. Taking running on a computer device as an example, Figure 1 This is a hardware structure block diagram of a computer device for an element verification method according to an embodiment of this application. For example... Figure 1 As shown, a computer device may include one or more ( Figure 1Only one is shown in the diagram. A processor 102 (which may include, but is not limited to, a microprocessor (MPU) or a programmable logic device (PLD)) and a memory 104 for storing data are also shown. The computer device may further include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the computer device described above. For example, the computer device may also include components that are more... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0025] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the element verification method in this embodiment. The processor 102 executes various functional applications and data processing by running the computer programs stored in the memory 104, thus implementing the aforementioned method. The memory 104 may include high-speed random access memory and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to computer devices via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0026] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by a communication provider for the computer equipment. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module used for wireless communication with the Internet.

[0027] This embodiment provides a method for verifying elements, applied to the aforementioned computer terminal. Figure 2 This is a flowchart of an element verification method according to an embodiment of this application, such as... Figure 2 As shown, the process includes the following steps:

[0028] Step S202: Determine the target element to be tested and the functional semantic information of the target element according to the test instructions of the target object;

[0029] Understandably, the system receives natural language test instructions from users, such as "click the like button for the 3rd video on the App's homepage," and performs semantic parsing using a large language model to extract the functional intent from the test instructions. It then identifies the abstract functional semantic information of the target element, including its operation type (such as click, input, swipe), its functional context (such as "video playback page" or "recommendation stream"), semantic attributes (such as "interactive button for expressing liking"), and constraints (such as "3rd" or "located within a video card"), thus forming structured functional semantic information.

[0030] Step S204: Detect the interface image of the target interface to obtain the set of elements on the interface image, and determine the feature map of each element in the set of elements.

[0031] It should be noted that the target element is located on the target interface.

[0032] Optionally, a visual acquisition strategy (such as capturing static pages or launching a screen mirroring program to capture real-time frames) is dynamically selected based on the element type of the target element and input into the large-scale visual model for processing. The model performs functional block causal segmentation, object detection, and optical character recognition (OCR) semantic extraction on the interface image to identify all visible UI elements on the interface and generate a feature map for each element. A schematic diagram of the feature map for each element is shown below. Figure 3 As shown, the feature map consists of two parts: stable features (such as the functional block to which the element belongs and the parent node structure) and variable features (such as text content, visual style, coordinate position, and icon shape). All features are automatically generated based on the image content and do not depend on any predefined attributes or manual annotations.

[0033] The interface images (including real-time frames from screen projection or static screenshots) acquired through perception are input into a pre-trained visual model, which automatically performs multi-dimensional semantic parsing: First, based on the interaction logic and functional dependencies between UI elements, causal segmentation of functional blocks is performed, dividing the interface into multiple functional units, such as "message list area", "input text box area", "send operation area", "navigation bar area", etc., and determining the interaction dependencies between each block (such as "the triggering of the send button depends on the input area not being empty"); Second, visible UI elements in the interface are located based on object detection algorithms, and embedded text content is extracted using OCR technology to determine visual and text features, thereby determining the stable and volatile features of each UI element.

[0034] Among them, stable features refer to attributes that remain relatively unchanged and stable during version iteration, including the parent node hierarchy of an element and its functional block (such as "located below the comment section" or "message sending area").

[0035] Volatile characteristics refer to dynamic attributes that are susceptible to interface redesigns, theme changes, or device adaptations, including the element's original ID, absolute / relative coordinates, displayed text content, and visual styles (such as icon shape, button color, font weight, and border style).

[0036] Based on the above information, a structured feature map is generated. The nodes in the feature map are: <element identifier, stable feature set, volatile feature set, functional block to which it belongs, and causal relationship between adjacent elements>.

[0037] Step S206: Determine the target feature map corresponding to the target element based on the functional semantic information and the feature map of each element, and determine the position information of the target element based on the target feature map;

[0038] Optionally, the functional semantic information extracted in step S202 is matched with the feature map of each element generated in step S204 to filter out the most matching target feature map. That is, the "target feature map" refers to the feature map that successfully matches the target element. During the matching process, a text disambiguation rule base and a historical knowledge base can be introduced to identify and eliminate semantic interference (such as multiple "like" buttons). Finally, the target feature map corresponding to the determined target element is output, and the actual position of the target element on the screen is obtained based on the coordinate information recorded in the target feature map.

[0039] Step S208: Determine the test operation based on the test instruction and the location information, and execute the test operation to verify the target element.

[0040] Optionally, the location coordinates of the target element can be combined with the operation type (such as "click" or "input") in the original natural language command to generate an executable control command (such as an ADB click command or a Selenium mouse operation), which is then sent to the test terminal for execution. After execution, the operation response results are collected to determine whether the target element has been correctly triggered (such as interface transition, state change, or value update), thereby achieving automated verification of the test behavior.

[0041] Through the above steps, the test instructions input by the target object are parsed to determine the functional semantic information of the target element. The interface image of the target interface is parsed to generate a feature map of each element in the element set on the target interface. In the matching phase, the functional semantic information is dynamically matched with each feature map in the element set to filter out the target element with the best semantic match, and the pixel coordinates of the target element are output according to the target feature map of the target element. This embodiment of the application does not rely on a preset coordinate mapping table of elements. Even if the interface layout is reconstructed, the element position is offset, the text content is changed, or the visual style is updated, the target element can still be located through the consistency between the feature map and the functional semantic information. This can solve the problem in related technologies where fixed element attributes rely on manual maintenance, and frequent manual maintenance of scripts is required during interface version iteration or layout adjustment, resulting in low testing efficiency.

[0042] Optionally, step S206 above can be implemented in the following manner: determining the text similarity between the functional semantic information and the text features of each element, and determining the visual similarity between the functional semantic information and each element, wherein the feature map of each element includes: the text features and visual features of each element; determining the first weight corresponding to the text similarity and the second weight corresponding to the visual similarity according to the element type of the target element, wherein the element type includes one of the following: element stable, element dynamic, and text ambiguous; determining the comprehensive similarity according to the text similarity, the first weight corresponding to the text similarity, the visual similarity, and the second weight corresponding to the visual similarity; and determining the target feature map corresponding to the target element according to the comprehensive similarity.

[0043] In this embodiment, the text similarity between the functional semantic information extracted from the test instruction and the text features of each candidate element in the interface, as well as the visual similarity between the text features and the visual features of the element, are calculated. Each element's feature map contains the text content and visual information extracted by OCR, such as color, shape, icon style, and layout context. The weights of text similarity and visual similarity are determined based on the type of the target element (stable, dynamic, or ambiguous). The text similarity is multiplied by a first weight, and the visual similarity is multiplied by a second weight; these are then weighted and summed to obtain a comprehensive similarity. The feature map of the element with the highest comprehensive similarity is selected as the target feature map corresponding to the target element, and the coordinates in the target feature map are output.

[0044] Optionally, successful matching data from different applications, versions, and scenarios can be stored in the historical knowledge base, and the optimal weights for various element types can be continuously adjusted through machine learning models. For example, if it is found that the visual feature stability of an app's "dynamic" button is greater than its textual stability across multiple versions, the visual weight of that app can be gradually increased from 0.7 to 0.85. An attention mechanism can also be added to focus on the most semantically relevant visual regions when calculating similarity; for example, "the 3rd" can guide the model to focus on detecting elements in the 3rd video card. In addition, when the overall similarity is lower than the confidence threshold, a resampling process can be triggered, such as switching to a higher resolution screenshot, enabling OCR enhancement mode, or calling feature templates from historical similar versions for matching.

[0045] Optionally, embodiments of this application provide a method for dynamically adjusting the weights of text similarity and visual similarity based on the element type of a target element, comprising: determining a first preset weight corresponding to the text similarity, and determining a second preset weight corresponding to the visual similarity; when the element type of the target element is the element stable type, increasing the first preset weight according to a first correction value to determine the first weight, and decreasing the second preset weight according to a second correction value to determine the second weight; when the element type of the target element is the element dynamic type, determining the first preset weight as the first weight, and determining the second preset weight as the second weight; when the element type of the target element is the text ambiguous type, decreasing the first preset weight according to a third correction value to determine the first weight, and increasing the second preset weight according to a fourth correction value to determine the second weight.

[0046] In this embodiment, initial preset weights are set for text similarity and visual similarity, respectively. When the target element type is element-stable, that is, the consistency of the target element's features in version iterations is high and its text semantic repetition rate is low, the preset weight of text is enhanced by a first correction value, while the preset weight of visual similarity is reduced by a second correction value. When the target element type is element-dynamic, it is considered that both text features and visual features are in a medium fluctuation range, and no weight adjustment is performed. When the target element type is text-ambiguous, multiple elements are identified to have highly similar or completely identical text content, and the text semantic discrimination ability is weak. In this case, the preset weight of text is reduced by a third correction value, while the preset weight of visual similarity is increased by a fourth correction value.

[0047] For example, if an app consistently displays the same text "Settings" in multiple versions, with a fixed position and icon across different themes, it learns that in "element-stable" scenarios, the text weight should be 0.8 and the visual weight 0.2. However, in another app, user feedback indicates that the visual style of the "Claim" button, even with identical text, significantly impacts click success rate. Therefore, in "text-ambiguous" scenarios, the visual correction value is increased to +0.5, and the text correction value is decreased to -0.4. Furthermore, if a positioning attempt fails due to improper weight settings, the deviation is recorded, and the correction values ​​for subsequent similar scenarios are fine-tuned through an online learning mechanism.

[0048] In this embodiment, the correction value is used to achieve accurate response to different element types, so as to significantly improve the positioning accuracy in complex and ever-changing mixed scenarios and effectively reduce the maintenance cost and failure rate of automated testing.

[0049] Optionally, before determining the text similarity and visual similarity weights of the target element, it is also necessary to determine the element type of the target element, including: determining the update frequency of the target element based on the historical version data of the target interface, determining the text repetition rate between the text information corresponding to the target element and the text information corresponding to other elements of the target interface, and identifying the target element in the target interface to determine the number of identifications corresponding to the target element; if the update frequency is less than a preset update frequency and the text repetition rate is less than a preset text repetition rate, the element type of the target element is determined to be the element stable type; if the update frequency is greater than or equal to a preset update frequency and / or the text repetition rate is greater than or equal to a preset text repetition rate, the element type of the target element is determined to be the element dynamic type; if the number of identifications is greater than or equal to a preset number of identifications, the element type of the target element is determined to be the text ambiguous type.

[0050] In this embodiment, historical version data of the target interface is queried to analyze the update frequency of the target element in past versions, such as whether the element frequently undergoes positional changes, style changes, or functional migrations. Simultaneously, the text information corresponding to the target element is extracted and globally compared with the text content of all other elements in the current interface to calculate the text repetition rate of the target element's text information, i.e., the proportion of times the text appears as an identifier for other elements in the current interface. Furthermore, preliminary visual scanning is used to identify the number of candidate instances of the target element in the current interface to determine if there are multiple elements with the same semantics but different positions. When the update frequency of the target element is lower than a preset threshold (e.g., no structural or style changes in the last three versions), and the text repetition rate with other elements in the current interface is lower than a preset value (e.g., lower than 10%), the element is determined to be "stable element." If the update frequency is higher than or equal to the threshold, or the text repetition rate is higher than or equal to the threshold, it is determined to be "dynamic element." If the number of target element instances identified in the current interface is greater than or equal to a preset threshold (e.g., ≥3), it is determined to be "ambiguous text."

[0051] The embodiments of this application realize the automated determination of element types, reducing reliance on human experience; secondly, through multi-dimensional and multi-indicator judgment, the misclassification rate is effectively reduced, avoiding the problem of misclassification due to a single indicator.

[0052] Taking a certain app as an example, the user command is: "Click the 'Recommended' button at the top." First, checking the app's version history reveals that the "Recommended" button has been located in the top navigation bar in the past four versions, with no changes in style or position, resulting in an update frequency of 0, which is below the preset threshold of 1.0. Simultaneously, analyzing the current interface text shows that "Recommended" appears only once, with a text repetition rate of 0%, also below the preset threshold of 10%. However, during the recognition phase, it was found that in addition to the top navigation bar, there is another "Recommended" channel tag at the top of the article list, resulting in two "Recommended" text instances being recognized, totaling 4, exceeding the preset threshold of 3. Since the "recognition count ≥ preset threshold" condition is met, this element is determined to be "textually ambiguous."

[0053] Optionally, before detecting and analyzing the interface image of the target interface, it is also necessary to select a corresponding visual acquisition strategy based on the determined element type, including: determining the element type of the target element, wherein the element type includes one of the following: element stable, element dynamic, and text ambiguous; if the element type is element dynamic or text ambiguous, acquiring video data of the target interface, identifying changing frames in the video data, and determining the changing frames as the interface image of the target interface; if the element type is element stable, acquiring a static interface image of the target interface, and determining the static interface image as the interface image of the target interface.

[0054] In this embodiment, upon receiving a natural language instruction, the target element is first determined to be one of the following types: element-stable, element-dynamic, or text-ambiguous, based on multiple indicators such as update frequency, text repetition rate, and recognition count. When the target element is determined to be element-dynamic or text-ambiguous, meaning the target element has a high-frequency update or semantic ambiguity risk, static screenshots are prone to missing key interaction states. Therefore, real-time video data of the target interface is obtained through a screen mirroring protocol (such as scrcpy or AirPlay), and based on inter-frame difference and semantic change detection algorithms, "change frames" where the interface content or structure changes are automatically identified, such as new content loading, button highlighting, animation pop-ups, and the appearance of new elements after list scrolling. Dynamic frames with causal relationships are used as input images for subsequent feature extraction. When the target element is determined to be element-stable, meaning the element features of the target element change less, a single frame of static interface image is captured without video capture to avoid unnecessary overhead and data transmission delay.

[0055] Optionally, in this embodiment, the acquisition method can be determined not only based on the element type, but also dynamically adjusted according to device performance, network bandwidth, and testing environment. For example, in low-performance devices or weak network environments, if the element is determined to be dynamic, the video sampling frequency can be reduced (e.g., from 30fps to 15fps), while still acquiring key change frames. In multi-screen collaboration scenarios, different acquisition channels can be allocated to different element types to achieve parallel acquisition and asynchronous processing. Furthermore, the timing of the next change can be predicted based on historical change patterns (e.g., a new pop-up always appears 1.2 seconds after a button is clicked), allowing for pre-loading and intelligent caching to further improve response efficiency. For text-ambiguous scenarios, a text disambiguation rule base can be simultaneously activated to compare the text distribution in the change frames with the disambiguation rules in real time, eliminating text ambiguity.

[0056] In this embodiment of the application, by only collecting key change frames, the storage and transmission overhead of video data is greatly reduced, CPU and memory usage is reduced, test execution efficiency is improved, and the problem of positioning failure caused by static screenshots failing to capture target elements (such as buttons that have not yet been loaded or briefly appearing loading states) in dynamic interfaces is avoided.

[0057] For example, a user inputs the command: "In the app, after swiping to the 5th video, click the 'Like' button." First, it's determined that the target element, the "Like" button, is dynamic (due to the high-frequency refresh of the recommendation feed and changes in button position) and poses a risk of textual ambiguity (multiple video cards have "Like" buttons). Therefore, screen mirroring is initiated, and a real-time video stream is captured, while simultaneously enabling change frame detection. When the 5th video finishes loading, the playback controls appear, and the video cover stabilizes, it's identified as a valid change frame, and this frame is extracted for feature extraction.

[0058] To better understand the above embodiments, such as Figure 4 As shown, the test scenario is classified into two levels of intelligent classification. Optionally, the first level is the terminal environment type, distinguishing between mobile terminals and PC Web terminals; the second level is the element type, which is divided into three categories: element stable type, which refers to low interface iteration frequency and text repetition rate below the preset threshold in multiple version iterations; element dynamic type, which refers to high interface iteration frequency in multiple version iterations, or high frequency of repeated occurrence of target element text in the current page; and text ambiguous type, which refers to the text description corresponding to the target element appearing multiple times in the current interface, which is easy to cause positioning confusion.

[0059] Based on the above two-level classification results, when the terminal is a mobile or PC web client and the element type is "dynamic element" or "textually ambiguous," a screen mirroring service (such as scrcpy) with a preset path is initiated, and a sampling scheme based on inter-frame semantic change detection is launched. This collects image frames in the interface that undergo causal structural changes (such as new content loading, interactive feedback pop-ups, or target elements entering the viewport after list scrolling), skipping redundant frames without changes to reduce bandwidth and computational load. Simultaneously, a text disambiguation rule library bound to the current application is loaded to provide constraints for the semantic resolution of ambiguous text. When the terminal is a mobile or PC web client and the element type is "stable element," an interface is called to obtain a static page screenshot; the screen mirroring process does not need to be initiated, saving resource overhead. For "textually ambiguous" scenarios, the application's text disambiguation rules need to be loaded first before visual acquisition to generate a contextual semantic graph.

[0060] Optionally, if there is no continuous operation for more than a preset time (e.g., 30 seconds), the screen projection process enters a low-power listening mode, retaining only the event wake-up capability; after the test command is executed, the process and device connection are released according to a preset strategy (e.g., "single command shutdown" or "keep connection to support continuous operation").

[0061] Optionally, after performing the test operation to verify the target element, the method further includes: determining whether the test operation was successfully executed; if the test operation fails, determining whether the location information is incorrect location information; and if the location information is incorrect location information, determining the reason for the positioning failure.

[0062] Optionally, after determining the reason for the location failure, the method further includes: if the reason for the location failure is text recognition failure, increasing the second weight corresponding to the visual similarity; if the reason for the location failure is visual recognition failure, determining the updated visual features of each element, and updating the feature map of each element according to the updated visual features of each element; if the reason for the location failure is block location recognition failure, determining the updated visual features of each element, and updating the feature map of each element according to the updated visual features of each element, wherein the updated visual features include: the updated functional block location features.

[0063] In this embodiment, the success of the test operation is determined as follows: if the target element is correctly clicked, input, or triggered, and the subsequent interface state meets expectations, the positioning is considered successful; if the operation is unresponsive, redirects to an error page, or the element is not recognized, the positioning is considered a failure, and it is determined whether the failure stems from a deviation in the output location information itself. If the location information is confirmed to be incorrect, the reasons for the positioning failure are further analyzed. The reasons for positioning failure are divided into three categories: text recognition failure, visual recognition failure, or block location recognition failure.

[0064] Once the cause of the location failure is determined, if it is a text recognition failure, meaning that the text content of the target element in the current interface is semantically consistent but is misjudged or not extracted correctly (such as OCR omission, blurry font, or mixed multilingual text), the second weight corresponding to visual similarity will be increased. That is, the weight of visual similarity of the element in the "element dynamic" and "text ambiguous" scenarios will be increased, while the weight of text similarity will be decreased. If it is a visual recognition failure, it means that the appearance features of the target element have shifted due to theme changes, icon updates, style reconstruction, etc. The visual attribute information of the element in the current interface will be re-collected, and new visual features will be generated by combining the deep feature extraction model and the feature map will be updated. If it is a block position recognition failure, it means that the causal segmentation rules of the functional block have failed. For example, if a button moves from the "information flow area" to the "toolbar area", the interaction dependency relationship between the element and its neighboring elements will be re-analyzed.

[0065] Optionally, when the number of failures of a certain type (such as the "share" button of an app failing multiple times due to visual drift in multiple updates) exceeds a preset threshold, a "high-risk element for visual drift" label can be generated. This allows for the collection of multiple sets of visual samples in advance in subsequent versions to establish a multimodal template library. Furthermore, user feedback or manual annotation can be combined to validate the automatic attribution results and improve attribution accuracy.

[0066] Through the embodiments of this application, the positioning capability is continuously optimized, and the positioning accuracy is improved after long-term use.

[0067] Optionally, such as Figure 5 As shown, a method for updating a knowledge base is presented, including:

[0068] Step S501: In the event of a location failure, calculate the text similarity deviation, visual feature deviation, and block position offset based on the current operation log and feature map.

[0069] Step S502: Determine whether the text similarity deviation is greater than the first threshold.

[0070] Step S503: If the text similarity deviation is less than or equal to the first threshold, determine whether the visual feature deviation is greater than the second threshold.

[0071] Step S504: If the visual feature deviation is less than or equal to the second threshold, determine whether the block position offset is greater than the third threshold.

[0072] Step S505: If the text similarity deviation is greater than the first threshold, it is determined to be the root cause of text ambiguity.

[0073] Step S506: Update the text disambiguation rule base for the root causes of text ambiguity, and increase the visual matching weight.

[0074] Step S507: If the visual feature deviation exceeds the standard, it is determined to be the root cause of visual variation.

[0075] Step S508: For the root cause of visual variation, reconstruct the visual feature template of the target element and integrate new visual features (such as icons and styles) into the variable feature set in the feature map.

[0076] Step S509: If the block position offset exceeds the limit, it is attributed to the layout offset root cause.

[0077] Step S510: Optimize the causal segmentation rules of functional blocks for the root cause of layout offset, and update the block affiliation relationship in the stable feature set.

[0078] Step S511: Write the attribution results and failure context together into the history knowledge base.

[0079] Optionally, after performing the test operation to verify the target element, the method further includes: obtaining multiple historical verification results of the target element, and determining the abnormal trend information of the target element based on the multiple historical verification results; predicting the abnormal probability of the target element based on the abnormal trend information; and, if the abnormal probability is greater than or equal to a preset abnormal probability, determining a correction strategy for the target element based on the element type of the target element and the abnormal probability, and correcting the target element according to the correction strategy.

[0080] In this embodiment of the application, after verifying the target element, the verification result is recorded in a historical knowledge base, and the abnormal trend information of the target element is determined based on the multiple historical verification results, wherein the abnormal trend information includes at least one of the following:

[0081] Visual similarity trend, which is the trend of visual embedding cosine similarity between the target element and the template at each match;

[0082] Text matching confidence trend, i.e., the trend of semantic similarity (such as BERTS score) and distance between the OCR recognition result and the expected semantics;

[0083] The layout offset trend is the trend of the relative coordinate offset of the target element in the functional block (such as the normalized position relative to the top left corner of the parent container).

[0084] Based on the above abnormal trend information, the probability of the target element becoming abnormal in the next version is determined. When the probability of an element becoming abnormal is high, a correction strategy is generated.

[0085] To better understand the process of the above-mentioned element verification method, the implementation flow of the above-mentioned element verification method will be described below in conjunction with optional embodiments, but it is not intended to limit the technical solution of the embodiments of this application.

[0086] This embodiment provides an element verification system. Figure 6 This is a schematic diagram of an element verification system according to an embodiment of this application, such as... Figure 6 As shown, it includes: intent parsing and scene classification module, scene-based perception and linkage module, multimodal feature map generation module, causal guidance matching module, execution and root cause tracing module, directed self-evolution module, and historical knowledge base.

[0087] This embodiment provides a method for verifying elements. Figure 7 This is a flowchart (II) of an element verification method according to an embodiment of this application, including:

[0088] Step S701: Receive the natural language test command input by the user, parse its intent through the large language model, and extract the functional semantics and expected operation type of the target element.

[0089] Step S702: Combining instruction keywords with application version data, element text repetition rate, and interface update frequency in the historical knowledge base, classify the current test scenario into two levels: Level 1 is terminal type (mobile / PC Web), and Level 2 is element feature type (stable element, dynamic element, ambiguous text).

[0090] Step S703: Based on the secondary scene classification results, select the visual acquisition strategy: for dynamic element / ambiguous text type, start screen projection and acquire changing frames, and load text disambiguation rules at the same time; for stable element type, only acquire static screenshots; for ambiguous text type, the corresponding disambiguation rule library needs to be loaded.

[0091] Step S704: Preprocess the acquired interface image, input it into the visual large model to perform functional block causal segmentation, object detection and OCR text extraction, and determine the visible elements and the semantics corresponding to the elements.

[0092] Step S705: Divide each element into stable features (parent node, functional block) and volatile features (text, coordinates, icon, color), and generate a multimodal feature map containing "element-stable features-volatile features-causal relationships".

[0093] Step S706: Dynamically match the natural language function semantics with the candidate elements in the feature map, dynamically adjust the weights of text and visual similarity according to the scene type, calculate the comprehensive matching degree, and output the localization coordinates.

[0094] Step S707: Send the corresponding operation command to the test terminal, execute the UI interaction action, collect the execution response data in real time, determine whether the operation is successful, and determine the validity of the positioning.

[0095] Step S708: If the location fails, call the causal reasoning module to calculate the text similarity deviation, visual feature deviation and block position offset, and determine the root cause of the failure as one of text ambiguity, visual variation or layout offset.

[0096] Step S709: Perform targeted updates based on root cause type;

[0097] For example, text ambiguity is addressed by optimizing disambiguation rules and increasing visual weight; visual variation is addressed by updating element visual features; layout offset is addressed by reconstructing the functional block affiliation; and if the positioning is successful, the current version features, weights, and coordinates are stored in the historical knowledge base.

[0098] Step S710: Store all updated data and execution logs in the historical knowledge base.

[0099] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the related technology, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0100] This embodiment also provides an element verification device. This system is used to implement the above embodiments and preferred embodiments, and will not be repeated as already described. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the system described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0101] Figure 8 This is a structural block diagram of an element verification device according to an embodiment of this application, such as... Figure 8 As shown, the device includes:

[0102] The first determining module 82 is used to determine the target element to be tested and the functional semantic information of the target element according to the test instructions of the target object;

[0103] The second determining module 84 is used to detect the interface image of the target interface to obtain the set of elements on the interface image and determine the feature map of each element in the set of elements.

[0104] The third determining module 86 is used to determine the target feature map corresponding to the target element based on the functional semantic information and the feature map of each element, and to determine the position information of the target element based on the target feature map;

[0105] The verification module 88 is used to determine the test operation according to the test instruction and the location information, and execute the test operation to verify the target element.

[0106] The system described above parses the test commands input by the target object to determine the functional semantic information of the target element. Subsequently, the interface image of the target interface is parsed to generate a feature map of each element in the element set on the target interface. During the matching phase, the functional semantic information is dynamically matched with each feature map in the element set to filter out the target element with the best semantic match, and the pixel coordinates of the target element are output based on its target feature map. This embodiment does not rely on a preset element coordinate mapping table. Even if the interface layout is reconstructed, element positions are shifted, text content is changed, or visual styles are updated, the target element can still be located through the consistency between the feature map and the functional semantic information. This solves the problem in related technologies where manually maintained fixed element attributes require frequent script maintenance during interface version iterations or layout adjustments, leading to low testing efficiency.

[0107] Optionally, the third determining module is used to determine the text similarity between the functional semantic information and the text features of each element, and to determine the visual similarity between the functional semantic information and each element, wherein the feature map of each element includes: the text features and visual features of each element; determining a first weight corresponding to the text similarity and a second weight corresponding to the visual similarity according to the element type of the target element; determining a comprehensive similarity according to the text similarity, the first weight corresponding to the text similarity, the visual similarity, and the second weight corresponding to the visual similarity; and determining the target feature map corresponding to the target element according to the comprehensive similarity.

[0108] Optionally, the third determining module is configured to determine a first preset weight corresponding to the text similarity and a second preset weight corresponding to the visual similarity; when the element type of the target element is the element stable type, the first preset weight is increased according to a first correction value to determine the first weight, and the second preset weight is decreased according to a second correction value to determine the second weight; when the element type of the target element is the element dynamic type, the first preset weight is determined to be the first weight, and the second preset weight is determined to be the second weight; when the element type of the target element is the text ambiguous type, the first preset weight is decreased according to a third correction value to determine the first weight, and the second preset weight is increased according to a fourth correction value to determine the second weight.

[0109] Optionally, the third determining module is configured to determine the update frequency of the target element based on historical version data of the target interface, and to determine the text repetition rate between the text information corresponding to the target element and the text information corresponding to other elements of the target interface, and to identify the target element in the target interface to determine the identification quantity corresponding to the target element; if the update frequency is less than a preset update frequency and the text repetition rate is less than a preset text repetition rate, the element type of the target element is determined to be the element stable type; if the update frequency is greater than or equal to a preset update frequency and / or the text repetition rate is greater than or equal to a preset text repetition rate, the element type of the target element is determined to be the element dynamic type; if the identification quantity is greater than or equal to a preset identification quantity, the element type of the target element is determined to be the text ambiguous type.

[0110] Optionally, the first determining module is used to determine the element type of the target element; if the element type is dynamic or ambiguous, it acquires video data of the target interface, determines the changing frames in the video data, and determines the changing frames as the interface image of the target interface; if the element type is stable, it acquires a static interface image of the target interface and determines the static interface image as the interface image of the target interface.

[0111] Optionally, the above device further includes: an optimization module, configured to determine whether the test operation was successfully executed; if the test operation failed, determine whether the location information is incorrect location information; and if the location information is incorrect location information, determine the reason for the positioning failure.

[0112] Optionally, the optimization module is configured to: increase the second weight corresponding to the visual similarity when the location failure is due to text recognition failure; determine the updated visual features of each element when the location failure is due to visual recognition failure, and update the feature map of each element based on the updated visual features of each element; determine the updated visual features of each element when the location failure is due to block location recognition failure, and update the feature map of each element based on the updated visual features of each element, wherein the updated visual features include: the updated functional block location features.

[0113] Optionally, the optimization module is configured to acquire multiple historical verification results of the target element, and determine the abnormal trend information of the target element based on the multiple historical verification results; predict the abnormal probability of the target element based on the abnormal trend information; and, if the abnormal probability is greater than or equal to a preset abnormal probability, determine a correction strategy for the target element based on the element type of the target element and the abnormal probability, and correct the target element according to the correction strategy.

[0114] It should be noted that the above modules can be implemented by software or hardware. For the latter, they can be implemented in the following ways, but are not limited to: all the above modules are located in the same processor; or, the above modules are located in different processors in any combination.

[0115] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above method embodiments when run.

[0116] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0117] Embodiments of this application also provide an electronic device, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0118] In one exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.

[0119] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above method embodiments.

[0120] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps in any of the above method embodiments.

[0121] Embodiments of this application also provide a computer program that includes computer instructions stored in a computer-readable storage medium; a processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the steps in any of the above method embodiments.

[0122] Specific examples in this embodiment can be found in the examples described in the above embodiments and exemplary implementations, and will not be repeated here.

[0123] Obviously, those skilled in the art should understand that the modules or steps of this application described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. They can be implemented using computer-executable program code, and thus can be stored in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those presented here, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, this application is not limited to any particular combination of hardware and software.

[0124] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the principles of this application should be included within the protection scope of this application.

Claims

1. A method for verifying an element, characterized in that, include: Determine the target element to be tested and the functional semantic information of the target element according to the test instructions of the target object; The interface image of the target interface is detected to obtain the set of elements on the interface image, and the feature map of each element in the set of elements is determined. Based on the functional semantic information and the feature map of each element, the target feature map corresponding to the target element is determined, and the location information of the target element is determined based on the target feature map; The test operation is determined based on the test instruction and the location information, and the test operation is executed to verify the target element.

2. The method according to claim 1, characterized in that, Determining the target feature map corresponding to the target element based on the functional semantic information and the feature map of each element includes: The textual similarity and visual similarity between the functional semantic information and each element are determined respectively. The first weight corresponding to the text similarity and the second weight corresponding to the visual similarity are determined based on the element type of the target element. A comprehensive similarity score is determined based on the text similarity, the first weight, the visual similarity, and the second weight. The target feature map corresponding to the target element is determined based on the comprehensive similarity.

3. The method according to claim 2, characterized in that, Determining the first weight corresponding to the text similarity and the second weight corresponding to the visual similarity based on the element type of the target element includes: Determine a first preset weight corresponding to the text similarity, and determine a second preset weight corresponding to the visual similarity; When the element type of the target element is element-stable, the first preset weight is increased according to the first correction value to determine the first weight, and the second preset weight is decreased according to the second correction value to determine the second weight; When the element type of the target element is dynamic, the first preset weight is determined as the first weight, and the second preset weight is determined as the second weight; When the element type of the target element is text ambiguity, the first preset weight is reduced according to the third correction value to determine the first weight, and the second preset weight is increased according to the fourth correction value to determine the second weight.

4. The method according to claim 2, characterized in that, Before determining the first weight corresponding to the text similarity and the second weight corresponding to the visual similarity based on the element type of the target element, the method further includes: The update frequency of the target element is determined based on the historical version data of the target interface, and the text repetition rate between the text information corresponding to the target element and the text information corresponding to other elements of the target interface is determined. The target element is identified in the target interface to determine the number of identifications corresponding to the target element. If the update frequency is less than the preset update frequency and the text repetition rate is less than the preset text repetition rate, the element type of the target element is determined to be element-stable. If the update frequency is greater than or equal to a preset update frequency, and / or the text repetition rate is greater than or equal to a preset text repetition rate, the element type of the target element is determined to be dynamic. If the number of identified elements is greater than or equal to the preset number of identified elements, the element type of the target element is determined to be text ambiguity.

5. The method according to claim 2, characterized in that, After performing the test operation to verify the target element, the method further includes: If the test operation fails, determine the reason for the failure. If the location failure is due to text recognition failure, increase the second weight corresponding to the visual similarity. If the location failure is due to visual recognition failure, determine the updated visual features of each element, and update the feature map of each element based on the updated visual features of each element. If the location failure is due to a failure to identify the block location, the updated visual features of each element are determined, and the feature map of each element is updated based on the updated visual features of each element. The updated visual features include the updated functional block location features.

6. The method according to claim 1, characterized in that, Before detecting the interface image of the target interface, the method further includes: Determine the element type of the target element, wherein the element type includes one of the following: element stable, element dynamic, or text ambiguous; When the element type is dynamic or ambiguous, the video data of the target interface is obtained, the changing frame is determined in the video data, and the changing frame is determined as the interface image of the target interface. When the element type is element-stable, a static interface image of the target interface is obtained, and the static interface image is determined as the interface image of the target interface.

7. The method according to claim 1, characterized in that, After performing the test operation to verify the target element, the method further includes: Obtain multiple historical verification results for the target element, and determine the abnormal trend information of the target element based on the multiple historical verification results; Predict the probability of an anomaly of the target element based on the anomaly trend information; If the anomaly probability is greater than or equal to a preset anomaly probability, a correction strategy for the target element is determined based on the element type of the target element and the anomaly probability, and the target element is corrected according to the correction strategy.

8. An element verification device, characterized in that, include: The first determining module is used to determine the target element to be tested and the functional semantic information of the target element according to the test instructions of the target object; The second determining module is used to detect the interface image of the target interface to obtain the set of elements on the interface image and determine the feature map of each element in the set of elements. The third determining module is used to determine the target feature map corresponding to the target element based on the functional semantic information and the feature map of each element, and to determine the position information of the target element based on the target feature map; The verification module is used to determine the test operation based on the test instruction and the location information, and execute the test operation to verify the target element.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein the program, when executed, performs the method of any one of claims 1 to 7.

10. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to execute the method of any one of claims 1 to 7 through the computer program.

Images