Automatic testing method and device and storage medium

Abstract

The invention discloses an automatic testing method and device and a storage medium. The method comprises the steps of obtaining a screenshot of a target interface; identifying interface components in the screenshot through a target detection model, and obtaining the category, first position information and visual features of each interface component; performing text detection and recognition on the screenshot, and extracting text blocks in the screenshot and second position information and text content of each text block; based on the first position information of the interface components and the second position information of the text blocks, determining one interface component matched with each text block; based on the interface component, the text block and the matching relationship between the text block and the interface component, constructing a semantic layout map; reasoning according to the semantic layout map to generate at least one user operation sequence; the sequence of user operations is performed to perform automated testing.

Description

Technical Field

[0001] This application relates to the field of automated software testing technology, specifically to an automated testing method, apparatus, and storage medium. Background Technology

[0002] Automated software testing is a crucial step in ensuring software quality and improving development efficiency. Current automated testing methods generally rely on the application's internal structure information to locate and manipulate UI elements. Specifically, testing tools interact with the application under test to obtain its internal UI hierarchy, such as the DOM tree for the web, the view tree or accessibility tree for mobile devices, and then parse out the fixed identifiers for each UI control, such as buttons, text boxes, and list items. Test scripts are written based on these predefined identifiers and simulate user actions such as clicks, inputs, and swipes by calling the APIs of underlying testing frameworks like Selenium, Appium, or UI Automator, thereby verifying the correctness of preset functionalities.

[0003] However, the test scripts are highly coupled with the internal implementation details of the application. Once the UI structure changes due to version updates (even if the visual appearance remains the same), the original control identifiers may become invalid, causing many test scripts to fail to execute. This requires manual relocation of elements and script updates, resulting in high maintenance costs and an exceptionally fragile test suite. Secondly, this method is difficult to effectively handle cross-platform applications or scenarios with dynamically generated interface content, because the UI control tree structure varies greatly across different platforms or with different data, making it difficult to adapt to cross-environment using a stable set of identifiers. More fundamentally, the existing method lacks the ability to understand the semantics of the interface. It can only mechanically execute preset scripts and cannot, like a human tester, understand the interface functions and interaction logic by comprehensively perceiving the visual components, text information, and their spatial layout relationships on the interface. Therefore, it cannot autonomously reason and generate test operation sequences that conform to business logic.

[0004] Therefore, how to break through the dependence on internal structure and enable automated testing to have semantic understanding capabilities based on vision and text, so as to achieve more robust and intelligent test automation, has become an urgent technical problem to be solved. Summary of the Invention

[0005] The purpose of this application is to provide an automated testing method, apparatus, and storage medium to solve the problems in existing automated testing methods, such as high vulnerability of test scripts, high maintenance costs, poor cross-platform and dynamic interface adaptability, and lack of intelligent operation generation capabilities based on semantic understanding, caused by excessive reliance on the internal UI structure information of applications.

[0006] To achieve the above objectives, the first aspect of this application provides an automated testing method, comprising: Get a screenshot of the target interface; The target detection model is used to identify the interface components in the screenshot and obtain the category, first position information and visual features of each interface component. Perform text detection and recognition on the screenshot, and extract the text blocks in the screenshot, as well as the second position information and text content of each text block; Based on the first position information of the interface component and the second position information of the text block, determine an interface component that matches each text block; A semantic layout graph is constructed based on UI components, text blocks, and the matching relationship between text blocks and UI components; Generate at least one user operation sequence based on semantic layout graph reasoning; Execute user action sequences to perform automated tests.

[0007] In this embodiment, obtaining the category, first location information, and visual features of each interface component includes: for each identified component, outputting the component category, location coordinate bounding box, and confidence score, and determining the component's location coordinates as the first location information; extracting features from the image region of each component to generate a corresponding visual feature vector; merging the detection boxes when multiple detection boxes correspond to the same component; calculating the average color and transparency of the component's image region when the component is disabled or semi-transparent, comparing it with a preset threshold, and marking the component's state; and analyzing the layer stacking relationship of the occluded component in the interface when one component is occluded by other components to determine whether the occluded component is at the top interactive layer.

[0008] In this embodiment of the application, determining whether an occluded component is at the top interactive layer includes: obtaining style information of the occluded component, the style information including the stacking order of the occluded component; determining that the occluded component is at the top interactive layer if the stacking order attribute of the occluded component indicates that the occluded component is at the top layer; and determining that the occluded component is at the top interactive layer if the position coordinates of the occluded component overlap with the coordinate set of user interaction events.

[0009] In this embodiment of the application, extracting text blocks and their second location information and text content from a screenshot includes: obtaining the bounding box of the location coordinates of each text region and determining the location coordinates of the text region as the second location information; extracting the text content of each identified text region; performing semantic correction on the text content based on a preset language model; and grouping and inferring the reading order of the corrected text blocks to generate structured text information.

[0010] In this embodiment of the application, determining an interface component that matches each text block based on the first position information of the interface component and the second position information of the text block includes: calculating the intersection-union ratio (IUR) between the second position information of each text block and the first position information of each interface component; if the IUR is greater than or equal to a preset threshold, directly associating the text block with the interface component with the largest IUR; if the IUR is less than the preset threshold, calculating the spatial distance between the text block and the interface component, and associating the text block with the interface component with the closest spatial distance.

[0011] In this embodiment, a semantic layout graph is constructed based on interface components, text blocks, and the matching relationship between text blocks and interface components. This includes: defining each interface component and each text block as a node in the semantic layout graph; creating and storing node attributes for each node, wherein the attributes of interface component nodes include at least one of component category, first position information, visual feature vector, and text feature vector obtained based on associated text blocks; the attributes of text block nodes include at least one of text content, second position information, and text feature vector extracted based on text content; calculating the spatial relative position between any two nodes based on the first and / or second position information; establishing an edge representing the spatial relationship between the corresponding two nodes when the spatial relative position satisfies a preset proximity condition; establishing an edge representing the logical relationship between associated text block nodes and interface component nodes based on the matching relationship between text blocks and interface components; and fusing visual feature vectors and text feature vectors to generate a unified feature representation for each node.

[0012] In this embodiment of the application, generating at least one user operation sequence based on semantic layout graph reasoning includes: inputting the semantic layout graph into a pre-trained graph neural network model; performing representation learning and reasoning on each node in the graph through the graph neural network model to predict a node access sequence that conforms to the target business process, wherein each node in the node access sequence corresponds to an interface component to be operated; determining the operation type and operation parameters of each interface component to be operated based on the node access sequence to generate a user operation sequence; wherein the operation type includes at least one of click, input, and swipe, and the operation parameters include the text content to be input or the swipe trajectory.

[0013] In this embodiment of the application, the method further includes: triggering a repair process when an operation failure caused by interface changes is detected during the execution of a user operation sequence; wherein the repair process includes: obtaining the latest screenshot of the current target interface and constructing a current semantic layout map based on the latest screenshot; comparing the current semantic layout map with the historical semantic layout map on which the failed step was based; calculating the matching score between each node in the historical semantic layout map and each node in the current semantic layout map, wherein the matching score is calculated by weighting at least one of the following: textual semantic similarity, visual feature similarity, positional distance, and component category similarity between nodes; determining the mapping relationship between nodes in the historical semantic layout map and nodes in the current semantic layout map based on the matching score; replacing the historical node pointed to by the failed step in the user operation sequence with the corresponding current node according to the mapping relationship to update the user operation sequence; and re-executing the previously failed step based on the updated user operation sequence.

[0014] In this embodiment of the application, the method further includes: setting a first threshold and a second threshold, wherein the first threshold is higher than the second threshold; performing a replacement operation when the matching score is higher than the first threshold; verifying the validity of the replacement operation sequence in a sandbox environment when the matching score is between the second threshold and the first threshold, and performing the replacement operation after successful verification; and generating a prompt message requiring manual confirmation when the matching score is lower than the second threshold.

[0015] A second aspect of this application provides an automated testing apparatus, the apparatus comprising: The memory is configured to store instructions; The processor is configured to retrieve instructions from memory and, when executing instructions, to implement any of the aforementioned automated testing methods.

[0016] A third aspect of this application provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, it implements the automated testing method described above.

[0017] The technical solution of this application acquires a screenshot of the target interface and uses object detection models and text recognition technology to extract the visual features and location information of interface components, as well as the content and position of text blocks. Then, based on spatial relationships, the text blocks are matched with interface components to construct a semantic layout graph that integrates visual, textual, and spatial logical relationships. Based on this graph, user operation sequences are generated to execute automated tests. This method transforms the driving basis of automated testing from traditional, volatile application internal structure information to stable, WYSIWYG visual semantic information, forming a set of automated testing methods based on visual semantic understanding. This fundamentally reduces dependence on internal implementation details, thereby significantly improving the robustness, maintainability, and intelligence level of test scripts and the testing process.

[0018] Other features and advantages of the embodiments of this application will be described in detail in the following detailed description section. Attached Figure Description

[0019] The accompanying drawings are provided to further illustrate the embodiments of this application and form part of the specification. They are used together with the following detailed description to explain the embodiments of this application, but do not constitute a limitation on the embodiments of this application. In the drawings: Figure 1 The illustration shows a flowchart of an automated testing method according to an embodiment of this application; Figure 2 This illustration schematically shows a diagram illustrating the functional implementation of a module in an automated testing apparatus according to an embodiment of this application. Figure 3 The diagram illustrates the internal structure of a computer device according to an embodiment of this application. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only for illustration and explanation of the embodiments of this application and are not intended to limit the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0021] It should be noted that the acquisition, transmission, storage, use, and processing of data in the technical solution of this application all comply with relevant laws and regulations. In the embodiments of this application, certain existing industry solutions such as software, components, and models may be mentioned. These should be considered exemplary, intended only to illustrate the feasibility of implementing the technical solution of this application, and do not imply that the applicant has already used or necessarily used such solutions.

[0022] Furthermore, if the embodiments of this application involve descriptions such as "first" or "second," these descriptions are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, features defined with "first" or "second" may explicitly or implicitly include at least one of those features. Additionally, the technical solutions of various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. If the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed in this application.

[0023] Figure 1 The illustration shows a flowchart of an automated testing method according to an embodiment of this application. Figure 1 As shown in the figure, this application provides an automated testing method, which may include the following steps.

[0024] Step 101: Take a screenshot of the target interface.

[0025] In one embodiment, capturing a screenshot of the target interface is achieved by calling the screen capture interface provided by the operating system or testing framework. For example, in web automation testing, the `save_screenshot()` method of Selenium WebDriver can be used; in mobile testing, the `get_screenshot_as_file()` method of Appium can be used; and in desktop applications, PyAutoGUI, the Windows API, or similar tools can be used to capture full-screen or window area screenshots. Selenium WebDriver is an open-source tool for automating web browser operations, supporting multiple programming languages ​​and mainstream browsers. It achieves automated testing or web scraping operations by simulating user behaviors such as clicking, inputting, and scrolling. Appium is an open-source, cross-platform mobile application automation testing framework that drives mobile devices or emulators through the WebDriver protocol to execute automated test scripts. PyAutoGUI is a cross-platform GUI automation library based on Python that can control mouse and keyboard operations and supports screenshotting and image recognition for automating desktop application operations, such as simulating clicks, inputting, and capturing screen areas. The Windows API is a set of low-level programming interfaces provided by Microsoft operating systems, allowing developers to call system functions such as window management, file operations, and screen capture.

[0026] In one embodiment, if the interface is dynamic, such as containing animations, carousels, loading states, etc., the system will wait for the interface to enter a stable state before taking a screenshot.

[0027] Step 102: Identify the interface components in the screenshot using the object detection model, and obtain the category, first position information and visual features of each interface component.

[0028] Object detection models are computer vision models used to locate and identify multiple target objects in an image. Common models include YOLO, Faster R-CNN, and SSD. Interface components refer to the basic interactive elements in a graphical user interface, such as buttons, input boxes, checkboxes, icons, and menus. Categories refer to the type labels of interface components, such as "button" or "input box."

[0029] In one embodiment, obtaining the category, first location information, and visual features of each interface component includes: for each identified component, outputting the component category, location coordinate bounding box, and confidence score, and determining the component's location coordinates as the first location information; extracting features from the image region of each component to generate a corresponding visual feature vector; merging the detection boxes when multiple detection boxes correspond to the same component; calculating the average color and transparency of the component's image region when the component is disabled or semi-transparent, comparing it with a preset threshold, and marking the component's state; and analyzing the layer stacking relationship of the occluded component in the interface when one component is occluded by other components to determine whether the occluded component is at the top interactive layer.

[0030] In this embodiment of the application, determining whether an occluded component is at the top interactive layer includes: obtaining style information of the occluded component, the style information including the stacking order of the occluded component; determining that the occluded component is at the top interactive layer if the stacking order attribute of the occluded component indicates that the occluded component is at the top layer; and determining that the occluded component is at the top interactive layer if the position coordinates of the occluded component overlap with the coordinate set of user interaction events.

[0031] In one embodiment, the system uses a pre-trained object detection model, such as YOLOv8, to infer from the input screenshot, outputting the category label, bounding box coordinates, and confidence score for each detected component. The bounding box coordinates are recorded as the first location information. RoI pooling or cropping is performed on the image region of each component, i.e., the region within the bounding box, and the input is fed into a pre-trained convolutional neural network (CNN) to extract visual features, outputting a fixed-dimensional visual feature vector. If multiple detection boxes correspond to the same component (i.e., the Intersection over Union (IoU) between boxes is higher than a threshold, such as 0.5), the Non-Maximum Suppression (NMS) algorithm is used to retain the box with the highest confidence, merging or discarding the rest. For components in a disabled or semi-transparent state, the average color (grayscale value) and transparency of their image region are calculated and compared with a preset threshold: if the average grayscale and transparency are both lower than the threshold, it is marked as "disabled." If the transparency is higher than the threshold, it is marked as "semi-transparent."

[0032] If a component is obscured by other components, the system analyzes the stacking relationship between that component and the other components. Specifically, it obtains the style information of the obscured component, especially the stacking order attribute that controls the display hierarchy. If the stacking order attribute value is the largest within the current view range, or explicitly indicates that the component is above all other components, then the obscured component is determined to be at the top interactive layer. When user interaction events, such as clicks or touches, occur as simulated or captured by the system, the system obtains the coordinate set corresponding to the user interaction event. If the first position information of the obscured component, i.e., its bounding box coordinates, overlaps spatially with the coordinate set of the user interaction event—for example, if the click coordinates happen to fall within its bounding box area—even if it is visually partially obscured, the system still determines that it is at the top interactive layer.

[0033] Step 103: Perform text detection and recognition on the screenshot, and extract the text blocks in the screenshot, as well as the second position information and text content of each text block.

[0034] In one embodiment, extracting text blocks and their second location information and text content from a screenshot includes: obtaining the bounding box of the location coordinates of each text region and determining the location coordinates of the text region as the second location information; extracting the text content of each identified text region; performing semantic correction on the text content based on a preset language model; and grouping and inferring the reading order of the corrected text blocks to generate structured text information.

[0035] In one embodiment, the system first scans the input screenshot using a text detection model to identify all regions that may contain text and obtains the bounding box coordinates of each text region as secondary location information. Next, a text recognition model is used to recognize characters in each identified text region, extracting the text content. Semantic correction is then performed on the recognized text content based on a preset language model to correct recognition errors caused by image quality or font style and to eliminate semantic ambiguity. Finally, layout analysis is performed on the corrected text blocks, grouping them according to their secondary location information, font size, and alignment, and inferring a reading order that conforms to human reading habits, thereby generating structured text information.

[0036] Step 104: Based on the first position information of the interface component and the second position information of the text block, determine an interface component that matches each text block.

[0037] In one embodiment, determining an interface component that matches each text block based on the first position information of the interface component and the second position information of the text block includes: calculating the intersection-union ratio (IUR) between the second position information of each text block and the first position information of each interface component; if the IUR is greater than or equal to a preset threshold, directly associating the text block with the interface component with the largest IUR; if the IUR is less than the preset threshold, calculating the spatial distance between the text block and the interface component, and associating the text block with the interface component with the closest spatial distance.

[0038] In one embodiment, for each text block, the intersection-union ratio (IUR) between its second location information and the first location information of each interface component is calculated. If the IUR of an interface component is greater than or equal to a preset threshold, the text block is directly associated with the interface component with the largest IUR as its matching component. If the IUR of all interface components is less than the preset threshold, it indicates that the text block and the interface components do not significantly overlap. In this case, the spatial distance, such as Euclidean distance, between the text block and each interface component is calculated, and the text block is associated with the interface component with the closest spatial distance to establish its semantic correspondence.

[0039] In one embodiment, to improve matching accuracy and handle edge cases, the following operations can also be performed: When associating based on spatial distance, it is further determined whether the interface component has a text container attribute, such as an input box, button, label container, etc. If it does, the association is accepted; otherwise, the text block is marked as an independent text node. For text blocks located at the intersection of multiple interface components or close to multiple components, the association is preferentially performed with components whose semantic category is "text label" or "input prompt".

[0040] Step 105: Construct a semantic layout graph based on interface components, text blocks, and the matching relationship between text blocks and interface components.

[0041] In one embodiment, a semantic layout graph is constructed based on interface components, text blocks, and the matching relationship between text blocks and interface components. This includes: defining each interface component and each text block as a node in the semantic layout graph; creating and storing node attributes for each node, wherein the attributes of interface component nodes include at least one of component category, first position information, visual feature vector, and text feature vector obtained based on associated text blocks, and the attributes of text block nodes include at least one of text content, second position information, and text feature vector extracted based on text content; calculating the spatial relative position between any two nodes based on the first and / or second position information of the nodes; establishing an edge representing the spatial relationship between the corresponding two nodes when the spatial relative position satisfies a preset proximity condition; establishing an edge representing the logical relationship between associated text block nodes and interface component nodes based on the matching relationship between text blocks and interface components; and fusing visual feature vectors and text feature vectors to generate a unified feature representation for each node.

[0042] In one embodiment, the system first defines each identified interface component and each extracted text block as an independent node in the semantic layout graph, and creates and stores node attributes for each node. The attributes of the interface component node include at least one of the following: component category, first position information, visual feature vector, and text feature vector generated based on the matched text block. The attributes of the text block node include at least one of the following: text content, second position information, and text feature vector extracted based on the text content. Subsequently, based on the first position information (for component nodes) and / or the second position information (for text nodes), the spatial relative position between any two nodes in the graph is calculated. If the spatial relative position of two nodes meets a preset proximity condition, such as their bounding box distance being less than a set threshold or overlapping, an edge representing the spatial relationship is established between the corresponding two nodes. This edge can be labeled with a specific orientation, such as "located above". Simultaneously, based on the determined matching relationship between the text block and the interface component, an edge representing a logical relationship, such as a "label correspondence" relationship, is established between each associated text block node and the interface component node. Finally, for each node, its visual feature vector (e.g., from image recognition) is fused with its text feature vector (e.g., from OCR and semantic embedding), and a unified feature representation of the node is generated through concatenation, weighting, or neural network encoding, which serves as the node embedding in the graph structure.

[0043] In one embodiment, to enhance the completeness and reasoning ability of the semantic layout graph, for pure icon-type components in the interface that are not associated with any text blocks, a similarity search can be performed between the component's visual feature vector and the historical component library in the graph database. The inferred semantic tags are then labeled for these components, and the components are incorporated as attributes into the nodes. For interface components with parent-child or container containment relationships, special edges representing these relationships are established based on the component's hierarchical structure. Simultaneously, version timestamps are appended to all nodes and edges to support subsequent interface change detection and graph version comparison. Here, container containment relationship refers to the visual or logical hierarchical relationship of inclusion and being included between components in the user interface.

[0044] In one embodiment, after each successful construction of a semantic layout graph, the system stores the graph structure and its node information in a graph database as a historical benchmark for subsequent interface change comparison and self-repair.

[0045] Step 106: Generate at least one user operation sequence based on semantic layout graph reasoning.

[0046] A user action sequence refers to a set of interface interaction instructions executed sequentially to achieve a specific business goal, such as logging in or submitting a form.

[0047] In one embodiment, the system generates at least one user operation sequence based on semantic layout graph reasoning, including: inputting the semantic layout graph into a pre-trained graph neural network model; performing representation learning and reasoning on each node in the graph through the graph neural network model to predict a node access sequence that conforms to the target business process, wherein each node in the node access sequence corresponds to an interface component to be operated; determining the operation type and operation parameters of each interface component to be operated based on the node access sequence to generate a user operation sequence; wherein the operation type includes at least one of click, input, and swipe, and the operation parameters include the text content to be input or the swipe trajectory.

[0048] In one embodiment, the system inputs the constructed semantic layout graph into a pre-trained graph neural network model. This model learns representations of each node and its connecting edges in the graph to understand the functions, states, and relationships of interface components. Next, the model infers based on specific business objectives, predicting an ordered sequence of node accesses. Each node in this sequence corresponds to an interface component to be operated on, and the sequence order reflects the reasonable operational steps required to complete the objective. The system determines the corresponding operation type and parameters for each interface component to be operated on based on its category, attributes, and context in the graph. For example, for "button" type nodes, the operation type is typically "click"; for "input box" type nodes, the operation type is "input," and specific text needs to be obtained from its associated text node or a pre-set data pool as the operation parameter; for "slider" or "list" type nodes, a "slide" operation and corresponding trajectory parameters may be generated. Finally, the system converts the node access sequence into a clear, executable user operation sequence, serving as the instruction basis for automated testing.

[0049] In one embodiment, when the confidence level of the node sequence output by the graph neural network model falls below a preset threshold, the system will fall back to a rule-based matching mode. This mode incorporates common business process templates, such as a login process typically consisting of "locating the account input box, entering the account, locating the password input box, entering the password, locating the login button, and clicking." The system generates the operation sequence by matching predefined pattern subgraphs within the graph structure. Furthermore, when generating operation parameters, for steps requiring text input, the system prioritizes text blocks directly associated with the component in the semantic layout graph, such as prompt text next to the input box. If no associated text is available, it selects text from a pre-configured test dataset. The final generated operation sequence is output as a structured instruction list.

[0050] Step 107: Execute the user action sequence to perform automated testing.

[0051] In one embodiment, the method further includes: triggering a repair process when an operation failure due to interface changes is detected during the execution of a user operation sequence; wherein the repair process includes: obtaining the latest screenshot of the current target interface and constructing a current semantic layout map based on the latest screenshot; comparing the current semantic layout map with the historical semantic layout map on which the failed step was based; calculating the matching score between each node in the historical semantic layout map and each node in the current semantic layout map, wherein the matching score is calculated by weighting at least one of textual semantic similarity, visual feature similarity, positional distance, and component category similarity between nodes; determining the mapping relationship between nodes in the historical semantic layout map and nodes in the current semantic layout map based on the matching score; replacing the historical node pointed to by the failed step in the user operation sequence with the corresponding current node according to the mapping relationship to update the user operation sequence; and re-executing the previously failed step based on the updated user operation sequence.

[0052] In this embodiment of the application, the method further includes: setting a first threshold and a second threshold, wherein the first threshold is higher than the second threshold; performing a replacement operation when the matching score is higher than the first threshold; verifying the validity of the replacement operation sequence in a sandbox environment when the matching score is between the second threshold and the first threshold, and performing the replacement operation after successful verification; and generating a prompt message requiring manual confirmation when the matching score is lower than the second threshold.

[0053] In one embodiment, the system executes an operation sequence by invoking the corresponding automated test-driven engine. During execution, the system monitors the result of each step in real time. When an operation fails due to a disappearance of UI elements, text changes, or layout adjustments, the system immediately pauses execution and triggers a repair process.

[0054] Specifically, the system acquires the latest screenshot of the current target interface and re-executes the aforementioned steps to quickly construct the current semantic layout graph. This graph is then compared with the historical semantic layout graph corresponding to the failed operation step. By calculating the matching score between each node in the historical graph and each node in the current graph—a score weighted by dimensions such as textual semantic similarity, visual feature similarity, positional distance, and component category similarity—the system assesses the degree of interface change. Based on the matching score, a graph matching algorithm determines the optimal mapping relationship between nodes in the two graphs. According to this mapping relationship, the system replaces the historical node pointed to by the failed step in the user operation sequence with the corresponding current node in the mapping relationship, thereby updating the user operation sequence. Finally, based on the updated sequence, the system re-executes the previously failed steps to attempt to continue the test process.

[0055] In this embodiment, to improve the accuracy and security of the repair, the method further includes: presetting two thresholds, namely a higher first threshold and a lower second threshold. When the calculated node matching score is higher than the first threshold, the system considers the matching confidence high and directly executes node replacement and operation update. When the matching score is between the second threshold and the first threshold, the system considers the matching to have some uncertainty, and places the updated operation steps in a sandbox environment to verify their effectiveness. Only after verification is the replacement applied in the actual test process. When the matching score is lower than the second threshold, the system considers the interface change too large and the automatic repair risk high. Therefore, it suspends automatic processing and generates a prompt message requiring manual confirmation, which is then submitted to the testers for review and decision-making. Sandbox verification is a risk management mechanism that pre-executes test steps in an isolated, controlled, and secure simulated environment to verify their correctness and security without directly affecting the real system or the main test process.

[0056] This application fundamentally solves the problem of large-scale test script failure caused by changes in interface layout, text, or style by transforming the traditional fragile location method based on fixed element identifiers into a context-aware location method based on joint modeling of image recognition and OCR. It also eliminates the inherent contradiction in traditional solutions where "test scripts are strongly coupled to interface implementation details." The system performs business process reasoning based on a semantic layout graph that integrates visual, textual, and layout information. It can automatically identify and generate operation sequences that conform to user intuition, eliminating the need for manually written or recorded scripts. This gives test cases semantic understanding aligned with business logic and cross-platform adaptability. Simultaneously, it introduces a test path self-repair mechanism based on graph structure comparison and semantic anchor matching, which can dynamically adjust test steps when local changes occur in the interface. This significantly reduces test maintenance costs and the frequency of manual intervention caused by application iterations, providing systematic support for achieving high coverage and low maintenance burden sustainable automated testing.

[0057] Figure 2 This diagram schematically illustrates the functional implementation of modules in an automated testing apparatus according to an embodiment of this application. Figure 2 As shown, it includes: S1, the interface screenshot acquisition layer, is responsible for capturing interface screenshots from terminals such as PC, mobile, Web, and App.

[0058] S2, the image analysis module, uses the "multi-scale target detection and recognition component" to perform image-level recognition on the screenshot.

[0059] The function of this module is to detect all interactive or visually meaningful interface components (such as buttons, input boxes, checkboxes, icons, menus, etc.) from the input screenshots and provide component location and category information for subsequent semantic analysis.

[0060] Input: Screenshot of the interface (RGB image). Output: List of interface components, including component ID, category label, location coordinates, confidence score, and optional visual feature vector.

[0061] S3, the structured OCR module, enables text detection, text recognition, and layout analysis, extracting and structuring text information from the interface.

[0062] This module is used to extract text information from interface screenshots and accurately associate it with identified components, thereby establishing a correspondence between text and visual elements.

[0063] Input: A screenshot of the same interface as the image analysis module. Output: A list of text blocks, including text ID, text content, location coordinates, recognition confidence score, reading order index, and the mapping relationship between text and components.

[0064] S4, the semantic layout graph construction module, abstracts interface elements into "nodes, components, edges, spaces, and logical relationships" to construct the semantic and layout structure of the interface.

[0065] The purpose of this module is to fuse visual information with OCR information to construct a semantic layout diagram, which is used to represent the spatial and logical relationships between components in the interface.

[0066] S5, the test path reasoning module, uses GNN and graph search algorithms to reason out the test paths that need to be executed.

[0067] This module is responsible for inferring executable user operation paths based on semantic layout graphs, thereby enabling the generation of automated test cases.

[0068] S6 has no code execution module; it automatically generates test instructions and calls the driver to execute tests.

[0069] This module is responsible for converting the inferred operation sequence into executable automated test scripts and executing them on the target application or webpage through a unified driver interface.

[0070] S7, the test path self-healing module, repairs problems in the test path through semantic anchors and graph alignment.

[0071] When the interface changes or the test steps fail, this module adaptively repairs the test path by matching and aligning it with the pre-stored login page component ID (a unique ID for each page) and data such as page layout and style with the graph structure, thus preventing script failure.

[0072] In a specific embodiment, the system acquires an RGB screenshot of the marketing system login page. After processing by the image analysis module, the screenshot outputs a list of components contained in the interface, specifically including: a "Username" text label, a "Please enter username" input box, a "Password" text label, a "Please enter password" input box, a "Login to System" button, and a "Register New Account" button. Taking the "Username" text label as an example, its output structured attributes include: Component ID: `baseInfo`; Category: `Label`; Position coordinates: bounding box coordinates are `{top: 50, right: 180, bottom: 100, left: 180}`; Confidence score: `95%`; Visual feature vector: represented as a tensor of size `[2, 512, 7, 7]`, which is extracted from the features of the original image through region pooling.

[0073] The system first inputs the entire screenshot into a pre-trained convolutional neural network to obtain the visual feature map of the whole image. Then, based on the coordinates of the detected component bounding boxes, the corresponding regions are located on the feature map, and a fixed-size visual feature representation is extracted using region pooling. Taking the "username" tag as an example, its corresponding region is pooled into a 512-channel, 7×7 feature tensor, which serves as the visual semantic embedding of this component.

[0074] The same screenshot of the login page is simultaneously input into the structured OCR module. The module executes the following process: 1. Text detection and recognition: Identify text blocks contained in the interface, such as: "Login" (title), "Username", "Password" (label text), "Please enter username", "Please enter password" (input box placeholders), "Login to system", "Register new account" (button text). Each text block is assigned a unique ID, text content, position bounding box, identification confidence level, and reading order index.

[0075] 2. Text-component association: The module associates the identified text blocks with the UI components output by the image analysis module. For example, it associates the "Username" text block with the "Username" label component, the "Please enter username" text block with the corresponding input box component, and the "Log in to system" text block with the login button component.

[0076] 3. Special text processing: For placeholder text such as "Please enter username", the system will combine its color characteristics, such as light gray, its location, such as being inside the input box, and the adjacent label text to assist in identification and association, ensuring that it is correctly classified as input prompt information rather than an independent label.

[0077] Based on the above analysis, the test path reasoning module automatically infers the user operation sequence required to complete the login process. This sequence is output in the form of a structured instruction list, for example: Operation 1: In the input box corresponding to the component "Please enter username", enter the test account, such as "xxx".

[0078] Step 2: In the input box corresponding to the component "Please enter password", enter a test password, such as "asb1234*".

[0079] Step 3: Click the button corresponding to the "Login System" component.

[0080] Each operation instruction clearly specifies the operation type, target component, and operation parameters. This sequence conforms to a typical login business process and can be directly handed over to the execution module for execution.

[0081] This application embodiment also provides an automated testing device, the device comprising: The memory is configured to store instructions; The processor is configured to retrieve instructions from memory and, when executing instructions, to implement any of the aforementioned automated testing methods.

[0082] This application also provides a machine-readable storage medium storing instructions that cause a machine to perform the above-described automated testing method.

[0083] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 3 As shown, the computer device includes a processor A01, a network interface A02, a display screen A04, an input device A05, and a memory connected via a system bus. The processor A01 provides computing and control capabilities. The memory includes internal memory A03 and a non-volatile storage medium A06. The non-volatile storage medium A06 stores an operating system B01 and a computer program B02. The internal memory A03 provides an environment for the operation of the operating system B01 and the computer program B02 stored in the non-volatile storage medium A06. The network interface A02 is used for communication with external terminals via a network connection. When the computer program is executed by the processor A01, it implements an automated testing method. The display screen A04 can be a liquid crystal display (LCD) or an e-ink display. The input device A05 can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device casing, or an external keyboard, touchpad, or mouse.

[0084] Those skilled in the art will understand that Figure 3The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0085] This application also provides a computer program product that, when executed on a data processing device, is suitable for executing a program that initializes an automated testing method.

[0086] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0087] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0088] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0089] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0090] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0091] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0092] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0093] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0094] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. An automated testing method, characterized in that, The method includes: Get a screenshot of the target interface; The target detection model is used to identify the interface components in the screenshot and obtain the category, first position information and visual features of each interface component. The screenshot is subjected to text detection and recognition to extract text blocks and the second position information and text content of each text block. Based on the first position information of the interface component and the second position information of the text block, determine an interface component that matches each text block; Based on the interface components, the text blocks, and the matching relationship between the text blocks and the interface components, a semantic layout diagram is constructed; At least one user operation sequence is generated based on the semantic layout graph; The user action sequence is executed to perform automated testing.

2. The automated testing method according to claim 1, characterized in that, The acquisition of the category, first location information, and visual features of each interface component includes: For each identified component, the component category, location coordinate bounding box, and confidence score are output, and the location coordinates of the component are determined as the first location information. Feature extraction is performed on the image region of each component to generate a corresponding visual feature vector; When multiple detection boxes correspond to the same component, the detection boxes are merged; When the component is disabled or semi-transparent, calculate the average color and transparency of the image area of ​​the component, compare it with a preset threshold, and mark the state of the component. If one of the components is obscured by other components, analyze the hierarchical stacking relationship of the obscured components in the interface to determine whether the obscured component is at the top interactive layer.

3. The automated testing method according to claim 2, characterized in that, Determining whether the occluded component is at the top level of interactivity includes: Obtain the style information of the occluded component, the style information including the stacking order of the occluded component; If the stacking order attribute of the occluded component indicates that the occluded component is at the top layer, then the occluded component is determined to be at the interactive top layer. If the position coordinates of the occluded component overlap with the coordinate set of user interaction events, the occluded component is determined to be at the top interactive layer.

4. The automated testing method according to claim 1, characterized in that, The extraction of text blocks and their second position information and text content from the screenshot includes: Obtain the bounding box of the position coordinates of each text region, and determine the position coordinates of the text region as the second position information; For each identified text region, extract the text content of the text region; The text content is semantically corrected based on a preset language model; The corrected text blocks are grouped and the reading order is inferred to generate structured text information.

5. The automated testing method according to claim 1, characterized in that, The step of determining an interface component that matches each text block based on the first position information of the interface component and the second position information of the text block includes: Calculate the intersection-union ratio (IUU) between the second position information of each text block and the first position information of each interface component; If the intersection-union ratio is greater than or equal to a preset threshold, the text block will be directly associated with the interface component with the largest intersection-union ratio; If the intersection-union ratio is less than a preset threshold, calculate the spatial distance between the text block and the interface component, and associate the text block with the interface component with the closest spatial distance.

6. The automated testing method according to claim 1, characterized in that, The step of constructing a semantic layout graph based on the interface components, the text blocks, and the matching relationship between the text blocks and the interface components includes: Each interface component and each text block are defined as a node in the semantic layout graph; Node attributes are created and stored for each node, wherein the attributes of the interface component node include at least one of component category, first position information, visual feature vector, and text feature vector obtained based on associated text block, and the attributes of the text block node include at least one of text content, second position information, and text feature vector extracted based on the text content. Based on the first and / or second location information of the nodes, calculate the spatial relative position between any two nodes; If the spatial relative positions satisfy a preset proximity condition, an edge representing the spatial relationship is established between the corresponding two nodes; Based on the matching relationship between the text block and the interface component, an edge representing the logical relationship is established between the associated text block node and the interface component node; The visual feature vector and the text feature vector are fused to generate a unified feature representation for each node.

7. The automated testing method according to claim 1, characterized in that, The step of generating at least one user operation sequence based on the semantic layout graph includes: The semantic layout graph is input into a pre-trained graph neural network model; The graph neural network model is used to learn and infer the representation of each node in the graph, and predict the node access sequence that conforms to the target business process. Each node in the node access sequence corresponds to an interface component to be operated. Based on the node access sequence, determine the operation type and operation parameters of each interface component to be operated, so as to generate the user operation sequence; The operation type includes at least one of clicking, inputting, and swiping, and the operation parameters include the text content to be input or the swiping trajectory.

8. The automated testing method according to claim 1, characterized in that, The method further includes: If an operation failure due to interface changes is detected during the execution of the user operation sequence, a repair process is triggered. The repair process includes: acquiring the latest screenshot of the current target interface and constructing a current semantic layout map based on the latest screenshot; comparing the current semantic layout map with the historical semantic layout map on which the failed steps were based; calculating the matching score between each node in the historical semantic layout map and each node in the current semantic layout map, wherein the matching score is calculated by weighting at least one of the following: textual semantic similarity, visual feature similarity, positional distance, and component category similarity between nodes; determining the mapping relationship between nodes in the historical semantic layout map and nodes in the current semantic layout map based on the matching score; replacing the historical node pointed to by the failed step in the user operation sequence with the corresponding current node according to the mapping relationship to update the user operation sequence; and re-executing the previously failed steps based on the updated user operation sequence.

9. The automated testing method according to claim 8, characterized in that, The method further includes: A first threshold and a second threshold are preset, wherein the first threshold is higher than the second threshold; If the matching score is higher than the first threshold, a replacement operation is performed; If the matching score is between the second threshold and the first threshold, the validity of the replacement operation sequence is verified in a sandbox environment, and the replacement operation is executed after the verification is successful. If the matching score is lower than the second threshold, a prompt message requiring manual confirmation is generated.

10. An automated testing device, characterized in that, The device includes: The memory is configured to store instructions; A processor is configured to retrieve the instructions from the memory and, when executing the instructions, to implement the automated testing method according to any one of claims 1 to 9.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the automated testing method as described in any one of claims 1 to 9.

Images