Automatic test method and equipment for user interface, storage medium and program product

Abstract

The embodiment of the invention provides a user interface automatic testing method and device, a storage medium and a program product. In the embodiment of the invention, a UI layout template of a user interface of which the layout is changed is obtained, and the template comprises a UI layout control, a to-be-tested field corresponding to the UI layout control and a constraint rule; matching a target data feature mode from a problem feature library according to the field type of the to-be-tested field; inputting a data feature rule in the target data feature mode, the to-be-tested field and a constraint rule of the to-be-tested field into a data generation model, and generating target test data by taking the constraint rule as a boundary condition and combining the data feature rule; and backfilling the target test data to the UI layout template to form instantiated UI layout data and loading the instantiated UI layout data so as to automatically test the user interface. According to the scheme, the historical abnormal knowledge is structured and is used for guiding the intelligent generation of the test data to automatically test the user interface, so that the test coverage and adaptability are improved, and the maintenance cost is reduced.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a user interface automated testing method, device, storage medium, and program product. Background Technology

[0002] UI (User Interface) layout issues refer to visual problems such as layout errors that occur during the rendering process of an app (application). Current layout detection achieves a degree of automation by writing automated scripts to simulate partial data input and combining this with screenshot comparison tools to identify interface changes.

[0003] However, such methods require the development of separate scripts for different layout anomalies, resulting in poor scalability and difficulty in adapting to application scenarios with frequent changes in interface structure. This leads to high maintenance costs and difficulty in guaranteeing detection coverage. Summary of the Invention

[0004] Embodiments of this application provide a user interface automated testing method, device, storage medium, and program product, which are used to automatically generate test data and perform tests for different layout anomalies, thereby effectively improving test coverage and adaptability and reducing maintenance costs.

[0005] This application provides an automated user interface testing method, comprising: obtaining a UI layout template of a user interface whose layout has changed, the UI layout template including UI layout controls and their corresponding fields to be tested and constraint rules; determining the field type of the field to be tested, and matching a target data feature pattern from a problem feature library according to the field type; the problem feature library includes multiple data feature patterns, each data feature pattern including at least one data feature rule, each data feature rule including a test data example and its triggering user interface exception reason information and exception problem description information; inputting the data feature rules in the target data feature pattern, the field to be tested and its constraint rules into a data generation model, using the constraint rules as boundary conditions, and generating target test data for the field to be tested according to the test data example in the data feature rules and its triggering user interface exception reason information and exception problem description information; backfilling the target test data into the UI layout template to obtain instantiated UI layout data; and loading the UI layout data through a UI automation tool to perform automated testing on the user interface.

[0006] This application also provides an electronic device, including a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor is able to implement the various steps in the user interface automated testing method provided in this application.

[0007] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, enables the processor to implement the various steps in the user interface automated testing method provided in this application.

[0008] This application also provides a computer program product, including a computer program / instruction, which, when executed by a processor, enables the processor to implement the various steps in the user interface automated testing method provided in this application.

[0009] In this embodiment, a UI layout template of the user interface whose layout has changed is obtained. This template includes UI layout controls and their corresponding fields to be tested and constraint rules. The field type of the field to be tested is determined, and a target data feature pattern is matched from the problem feature library based on the field type. The data feature rules in the target data feature pattern, the field to be tested, and its constraint rules are input into the data generation model. Using the constraint rules as boundary conditions, target test data is automatically generated by combining test data examples in the data feature rules and their triggering user interface anomaly cause information and anomaly problem description information. The target test data is then backfilled into the UI layout template to form instantiated UI layout data. The instantiated data is loaded through a UI automation tool to execute automated testing of the user interface. This solution achieves automated testing of the user interface by structuring historical anomaly knowledge and using it to guide the intelligent generation of test data, thereby effectively improving test coverage and adaptability and reducing maintenance costs. Attached Figure Description

[0010] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1a A flowchart illustrating an exemplary embodiment of this application for an automated user interface testing method; Figure 1b A schematic diagram of the model architecture of an automated user interface testing method provided for an exemplary embodiment of this application; Figure 2 A schematic diagram of the structure of an electronic device provided for an exemplary embodiment of this application. Detailed Implementation

[0011] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0012] It should be noted that, in the cases involving user information in the embodiments of this application, the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in the embodiments of this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse. In addition, the various models involved in this application (including but not limited to language models or large models) comply with relevant laws and standards.

[0013] To address the shortcomings of existing automated testing methods, such as the need to develop separate scripts for different UI layout anomalies, poor scalability, difficulty in adapting to frequent interface changes, high maintenance costs, and low detection coverage, this application proposes a solution. This involves obtaining a UI layout template of the user interface whose layout has changed. This template includes UI layout controls and their corresponding fields to be tested, along with constraint rules. The field type of the field to be tested is determined, and a target data feature pattern is matched from a problem feature library based on the field type. The data feature rules from the target data feature pattern, the field to be tested, and its constraint rules are input into a data generation model. Using the constraint rules as boundary conditions, and combining test data examples from the data feature rules with information on the reasons for triggering the user interface anomaly and the description of the anomaly, target test data is automatically generated. The target test data is then populated back into the UI layout template to form instantiated UI layout data. Finally, the instantiated data is loaded using a UI automation tool to execute automated testing of the user interface. This solution achieves automated testing of the user interface by structuring historical anomaly knowledge and using it to guide the intelligent generation of test data, thereby effectively improving test coverage and adaptability while reducing maintenance costs.

[0014] The technical solutions provided by the various embodiments of this application are described in detail below with reference to the accompanying drawings.

[0015] Figure 1a This is a flowchart illustrating an automated user interface testing method provided for an exemplary embodiment of this application. The method can be based on... Figure 1b The schematic diagram shown below illustrates the model architecture implementation of the user interface automated testing method provided in an exemplary embodiment of this application, as follows: Figure 1b As shown, the model architecture includes: a problem feature library, a data generation model, a multimodal analysis model, a UI automation tool, and a UI layout template optimization model. Figure 1a As shown, the method includes: S101. Obtain the UI layout template of the user interface whose layout has changed. The UI layout template contains UI layout controls and their corresponding fields to be tested and constraint rules. S102. Determine the field type of the field to be tested, and match the target data feature pattern from the problem feature library according to the field type. The problem feature library includes multiple data feature patterns, each data feature pattern includes at least one data feature rule, and each data feature rule includes a test data example and its triggering user interface exception reason information and exception problem description information. S103. Input the data feature rules, the field to be tested and its constraint rules in the target data feature pattern into the data generation model. Using the constraint rules as boundary conditions, generate target test data for the field to be tested based on the test data examples in the data feature rules and the information on the reasons for triggering user interface anomalies and the description of the anomalies. S104. Fill the target test data back into the UI layout template to obtain the instantiated UI layout data; load the UI layout data through the UI automation tool and perform automated testing on the user interface.

[0016] In this application embodiment, the user interface serves as the medium for information exchange and interaction between the application and the user, and is a visual presentation layer of the application's functions and content. An application may contain one or more user interfaces, through which the user perceives information, inputs data, and triggers the application's functional logic. This application embodiment does not limit the specific type of application; for example, it may be an App (Application) running on a mobile device, such as social media, online shopping, news, or utility applications; or it may be desktop software or Web (WorldWide Web) applications running on a personal computer, such as office software, design tools, or online service platforms.

[0017] Accordingly, the embodiments of this application do not limit the specific type of user interface. For example, a user interface may be a registration / login page containing form input controls, an e-commerce homepage displaying a product list, a backend management panel containing complex charts and operation controls, or various dialog boxes, settings menus, etc. The methods described in the embodiments of this application aim to automate the testing of any type of user interface that may undergo layout changes.

[0018] In this embodiment, step ① obtains the UI layout template of the user interface whose layout has changed. The UI layout template is templated data describing the layout structure of the user interface, used to define the organization method of interface elements and test parameters. This embodiment does not limit the specific implementation form or data structure of the UI layout template. For example, the UI layout template can be a configuration file in JSON (JavaScript Object Notation) format, or an XML (Extensible Markup Language) document, a YAML (Yet Another Markup Language or YAML Ain't Markup Language) file, or a data structure in memory, as long as it can completely describe the control layout and test constraints.

[0019] The UI layout template includes UI layout controls and their corresponding fields to be tested and constraint rules. UI layout controls refer to visual or interactive elements in the user interface. In this embodiment, the specific type of UI layout control is not limited. For example, a UI layout control can be a button, input box, text label, drop-down menu, or image container, etc. The field to be tested refers to the attribute or content of the UI control into which test data needs to be injected. In this embodiment, the type of field is not limited. For example, the field type can be the input value of an input box, the display text of a text label, the resource path of an image control (i.e., string, text, numeric), or an image URL (Uniform Resource Locator), etc. Constraint rules refer to the restrictions on the field to be tested, such as the maximum / minimum input length, data type (e.g., number, string, email format), allowed character sets, or business logic rules (e.g., "username must start with a letter").

[0020] In this embodiment, the UI layout template may also include the hierarchy of UI controls, size attributes (such as width and height), margins (such as margin and padding), and metadata such as text content templates and / or image link templates, which can collectively define the rendering logic of the interface. The control hierarchy can describe the parent-child nesting relationship of UI controls, such as a form container containing multiple input child controls; size attributes and margins can define the visual size and spacing rules of UI controls. The text content template is a sub-template of the UI layout template, which can be used to dynamically generate placeholder strings for text content. For example, "Welcome, {user_name}", when test data is filled in, the placeholder {user_name} will be replaced by the actual generated test data. The image link template is another sub-template of the UI layout template, which can be used to dynamically load parameterized URLs of image resources. For example, "https: / / cdn.example.com / images / {product_id}.jpg". During instantiation, {product_id} can be automatically filled with specific data values ​​based on the test data. This metadata ensures that the UI layout template not only guides the generation of test data, but also accurately reproduces the rendering effect of the user interface in automated tests.

[0021] In this embodiment of the application, the problem feature library is a dynamically constructed knowledge base used to store abnormal feature data extracted and structured from historical UI layout anomaly cases. The problem feature library can adopt a hierarchical organization system, with the top level based on field type division, each field type node attaching a corresponding data feature pattern, and each data feature pattern further containing multiple data feature rules, forming a three-level hierarchical structure of "field type → data feature pattern → data feature rule".

[0022] Data feature patterns are the core units in the problem feature library, categorized and organized for specific field types. For example, string types correspond to text input boxes, and numeric types correspond to monetary input boxes. Each data feature pattern includes at least one data feature rule. For instance, for the data feature pattern of string type fields, all anomalies related to text input are aggregated, covering problem scenarios such as boundary overflow and formatting errors.

[0023] Data feature rules are concrete instances of data feature patterns, describing fine-grained rules that depict how specific test data triggers UI anomalies. Each data feature rule includes three core parts: test data examples, information on the reason for triggering the user interface anomaly, and a description of the anomaly. Test data examples refer to specific data values ​​or data templates with clear characteristics that have actually triggered UI layout anomalies in historical testing, providing seeds or templates for generating new test data. For example, for a text field, an example could be "AAAAA" representing a continuous sequence of full-width characters; for an image URL field, an example could be "https: / / example.com / image_2000x10.png" representing a link to an image with an extremely large width and extremely small height. The information on the reason for triggering the user interface anomaly explains why the above test data examples cause user interface anomalies from a technical or layout logic perspective. This information is crucial for guiding the data generation model to understand the essence of the problem and thus generate similar variant test data. For example, regarding the full-width character example above, the reason could be: "Some layout engines do not accurately calculate the width of consecutive full-width characters, resulting in line break logic errors." Regarding the extremely wide image example, the reason could be: "The image's aspect ratio is extreme, the container height is fixed and not properly adapted, causing the image to stretch or squeeze, disrupting the layout flow." The anomaly description information specifically describes the observable visual or interactive malfunctions presented to the user when the user interface is rendered using this test data example. For example, regarding the reasons above, the problem description information could be: "Text is truncated at unexpected positions or produces extra whitespace, causing text overlap or abnormal control spacing," and "The image, after loading, breaks the predetermined layout, causing the content area below to be obscured or the overall layout scrolling to be abnormal."

[0024] For example, the problem feature library could contain a data feature pattern for text type fields. This pattern contains two data feature rules: The test data example for the first data feature rule is "AAAAAAAAAAAAAAAA", and the exception reason information is "In a fixed-width text container, when encountering a long string without spaces, the default line break algorithm of the mobile native rendering engine may not be able to find a suitable breakpoint, causing the calculated width of the text to exceed the container constraints". The problem description information is "The text content overflows its parent container boundary horizontally, obscuring the adjacent right-side controls or content, and disrupting the overall layout alignment". The test data example for the second data feature rule is "Hello\n\n\n\n\n\nWorld", and the exception reason information is "Multiple consecutive line breaks are parsed as requiring additional line height in the text rendering control. If the container height is dynamically calculated or there is a maximum height limit, the accumulated line height may cause the container to be abnormally stretched". The problem description information is "The height of the TextView container is far greater than expected, squeezing subsequent content on the user's page, causing scrolling errors or the bottom content being pushed out of the visible area". These data feature rules are all derived from the structured storage of actual UI layout anomaly cases, ensuring that the problem feature library retains historical experience and has generalization capabilities.

[0025] In this embodiment, determining the field type of the field to be tested can be achieved by parsing the control properties in the UI layout template to determine the precise field type. For example, when the UI layout template is in JSON format, parsing the properties node of the input UI layout control (such as a text box) typically includes fields such as type, inputType, or a custom dataType that identify the semantic function of the control. By reading the values ​​of these fields, the UI layout control can be mapped to the precise field type. For another example, if the UI layout control type is "ImageView" and its properties include "srcTemplate", then the precise field type of the field to be tested is determined to be "image URL".

[0026] Furthermore, such as Figure 1b Referring to step ②, target data feature patterns can be matched from the problem feature library based on the field type of the field to be tested. This process is dynamic and semantically driven. In this embodiment, the specific implementation of matching target data feature patterns from the problem feature library based on the field type of the field to be tested is not limited.

[0027] Optionally, this process can employ a combination of type inheritance fallback mechanism and semantic similarity calculation. Specifically, it matches predefined field type labels in the problem feature library; if no match is found, it falls back; simultaneously, it calculates the similarity between the semantic features of the field to be tested and the data feature patterns in the problem feature library, for example, using TF-IDF (Term Frequency-Inverse Document Frequency) vectorization to calculate cosine similarity, and selecting data feature patterns whose similarity meets the threshold as target data feature patterns. For example, when the field to be tested is of type "user_name", if there is no direct match in the problem feature library, the system calculates the similarity between "user_name" and labels in the problem feature library, such as "username" and "nickname". If the similarity of the "username" label meets the similarity threshold, it matches the data feature pattern corresponding to "username"; if all similarities do not meet the threshold, a default mode, such as the general "string" data feature pattern, can be enabled.

[0028] like Figure 1b Referring to step ③, the data feature rules, the field to be tested, and its constraint rules in the target data feature pattern can be input into the data generation model. Then, step ④ is executed to generate target test data for the field to be tested based on the constraint rules as boundary conditions, the test data examples in the data feature rules, the information on the reasons for triggering user interface anomalies, and the description of the anomalies.

[0029] Among them, the data generation model is based on a rule engine and a generative artificial intelligence model. Its core function is to receive data feature rules from the problem feature library and constraint rules from the UI layout template. Under dual guidance, it automatically generates target test data for the field to be tested, ensuring that the generated target test data retains the abnormal features of the UI layout abnormal cases and meets the requirements of business logic (i.e. constraint rules), rather than simply copying historical samples and test data examples.

[0030] Furthermore, the data generation model uses constraint rules as boundary conditions, meaning that the constraint rules defined in the UI layout template are used as hard or soft boundaries in the generation process. During test data generation, it ensures that all generated test data adheres to the predefined limitations of the field under test. For example, when the constraint rule is "username length ≤ 50 characters," the data generation model will dynamically suppress outputs exceeding 50 characters or automatically truncate excessively long results during the generation process, ensuring that all generated target test data is within the 50-character boundary, thus accurately verifying the robustness of the user interface under boundary conditions.

[0031] The target test data is the specific data instance that the data generation model ultimately outputs and is used to populate the UI layout template. It is new test data that cleverly integrates problem characteristics while satisfying all constraint rules. For example, the field to be tested is a "Username" input box with the constraint rules {Type: String; Maximum Length: 20 characters; Allowed Characters: Chinese and English, Numbers, Underscores}. The data feature rule 1 in the target data feature pattern is: "Test data example: User_Name_1234567890; Triggering reason information: When the username character length is close to or equal to the maximum length allowed by the control, under different fonts, font sizes, or rendering engines, the actual pixel width of the text may exceed the preset layout width of the container; Problem description information: The text is truncated at the end or overflows the container horizontally." This leads to incomplete visual information or layout disruption. Data feature rule 2 is: "Test data example: Zhang_ABC_Li_123; Triggering reason information: When a string with mixed character widths is automatically wrapped or aligned, the layout engine generates incorrect wrapping positions or spacing due to inconsistent character width calculations; Problem description information: The text wraps abnormally within the container, causing a surge in line height, or overlapping or misaligning with other elements in the same line." The generated target test data can be "User_Name_2345167819" and "A_Test_User_2024".

[0032] like Figure 1b Refer to step ⑤ to backfill the target test data into the UI layout template, resulting in instantiated UI layout data. Backfilling refers to the process of "filling" or "binding" the generated target test data into the corresponding positions in the UI layout template, based on the predefined correspondence between UI controls and fields to be tested. Instantiated UI layout data refers to a complete, executable interface description dataset formed by "filling" or "binding" the original UI layout template (containing placeholders and constraint rules) with specific target test data generated by the data generation model. This dataset fully defines all UI control properties and content, directly drives user interface rendering, or is used for test execution—a concrete data state.

[0033] Further, step ⑥ loads UI layout data using a UI automation tool to perform automated testing of the user interface. The instantiated UI layout data is passed as input to the UI automation tool, which parses the data and dynamically builds a test environment, then automatically executes a preset test process. The UI automation tool refers to a software testing framework or platform capable of simulating user interaction behavior, performing interface operations, and verifying rendering results; its core functions include loading UI layout data, driving interface rendering, and executing predefined operation sequences (such as clicks, swipes, and input). This application does not limit the specific type of UI automation tool; any tool that supports programmatically loading UI layout data and executing test processes is applicable. For example, a UI automation tool could be the cross-platform Appium (supporting iOS / Android applications), the web-based Selenium (for browser interface testing), platform-native tools such as Espresso (Android) and XCTest (iOS), cloud testing platform tools such as BrowserStack and Sauce Labs, or even a custom lightweight test script engine (such as a Puppeteer-based Headless Chrome driver).

[0034] The core function of generating target test data is to achieve anomaly-oriented and precise testing. On the one hand, by integrating UI layout anomaly case knowledge (i.e., data feature rules) with real-time constraint rules, high-coverage target test data is automatically generated, significantly reducing the cost of manual script development while dynamically adapting to UI layout changes. On the other hand, the generated target test data strictly adheres to business boundaries, avoiding interference from invalid test data, improving test accuracy, and ultimately achieving improved testing efficiency, reduced maintenance costs, and enhanced coverage, providing sustainable automated assurance for frequently iterating applications.

[0035] In one optional embodiment, UI layout anomaly cases are obtained. These cases include recorded user interface anomaly details, which at least include an abnormal image of the user interface, an abnormal UI layout control and its corresponding abnormal fields, abnormal field data, anomaly cause information, and anomaly problem description information. Following the order of UI control positioning, abnormal field association, information extraction, and structured encapsulation, and according to a preset structured field system, the abnormal image, abnormal UI layout control, abnormal fields, abnormal field data, anomaly cause information, and anomaly problem description information are structured to obtain structured interface anomaly information. The preset structured field system is hierarchical. Cross-case aggregation analysis is performed on the structured interface anomaly information to obtain various data feature rules. Based on the field type of the abnormal fields, these various data feature rules are categorized to obtain a problem feature library including at least one data feature pattern.

[0036] An abnormal image is a screenshot captured when a layout anomaly occurs in the user interface, visually recording the abnormal problem. For example, a screenshot showing severe overlap between product title text and adjacent price tags. An abnormal UI layout control is the interface element object identified in the abnormal image as directly causing or exhibiting the anomaly. It's not just a visual area, but also a locatable control instance in the program. For example, the TextView control displaying the overlapping product title in the screenshot above. An abnormal UI layout control and its corresponding abnormal field are data-bound items belonging to the aforementioned abnormal UI layout control that specifically caused the anomaly. It can be a settable property of the UI layout control. For example, for a TextView control, its abnormal field is the `text` (text content) property. The abnormal field data is the specific data value actually assigned to the abnormal field when a layout anomaly occurs, and is the direct cause of the layout anomaly. For example, the product title data causing text overlap could be: "2024 New Spring Limited Edition Ultra-Lightweight Breathable Sports Shoes (Black and White Available)". The anomaly cause information refers to the technical analysis of the root cause of the anomaly recorded from the perspective of developers or testers, explaining "why" that the anomaly field data caused the UI layout control to have an abnormal problem. For example, the title text is too long, and its parent container LinearLayout's width is set to match_parent, but the actual required width when containing long text is not correctly calculated, causing the text view width to overflow. The anomaly problem description information describes the final manifestation of the anomaly from the perspective of user experience or test verification, defining "what kind" phenomenon is considered a problem. For example, the product title text overflows to the right, overlapping with the price tag view on the right, causing the price information to be partially obscured.

[0037] In this embodiment, the aforementioned exception details are transformed according to the processing order of UI control positioning, exception field association, information extraction, and structured encapsulation, and a preset hierarchical structured field system. Specifically, UI control positioning is performed first, using automated tools to parse the exception image and extract the unique identifier and spatial coordinates of the exception UI layout control; secondly, exception field association is performed, that is, the exception UI layout control is mapped to the exception field; information extraction is performed, extracting the exception field data, exception cause information, and exception problem description information corresponding to the above-mentioned exception fields from the exception details information; structured encapsulation is performed, that is, according to the preset hierarchical structured field system: case identifier → user interface identifier → exception UI layout control path → exception field name → {exception field data, exception cause information, exception problem description information}, structured processing is performed to obtain structured interface exception information. Structured interface exception information includes: Case identifier: a unique number for exception cases, used for tracking and management; User interface identifier: the logical name or code path of the interface or page where the exception occurred; Exception UI layout control path: complete hierarchical location information from the root node of the interface to the specific exception control (such as XPath, view tree path or resource ID chain); Exception field name, exception field data, exception cause information, and exception problem description information.

[0038] For example, a UI layout anomaly case is an example of a layout anomaly occurring on the "Publish Product Review" page of an e-commerce application. The recorded user interface anomaly details are as follows: a long string of text without spaces was typed in the review input box: "This dress is really super beautiful, the fabric is very comfortable, the cut is also very good, the color is exactly the same as the picture, the logistics speed is also amazing, I will continue to repurchase in the future." After clicking publish, the submit button below the input box was pushed off the screen, and a screenshot attachment was also included. After the above process, the structured interface anomaly information obtained is: Case ID: UI-DEFECT-2024-0510-003, User Interface ID: com.example.shop / .ReviewPublishActivity, Abnormal UI layout control path: / android.widget.FrameLayout[0] / android.widget.LinearLayout[1] / android.widget.ScrollView[0] / android.widget.EditText[@resource-id=' [comment_input'], Abnormal field name: text, {Abnormal field data: This dress is really super beautiful, the fabric is very comfortable, the cut is also very good, the color is exactly the same as the picture, the logistics speed is also amazing, I will continue to repurchase in the future; Abnormal reason information: The EditText control received an excessively long text without spaces in multi-line mode. When the scrolling logic of its parent container ScrollView is not synchronized with the dynamic calculation of the internal layout height, the height of the container content is abnormally expanded, pushing the bottom fixed button out of the visible area; Abnormal problem description information: The submit button below the product review input box is completely squeezed out of the bottom visible area of ​​the screen, the user cannot click it, and the function is invalid}.

[0039] Furthermore, cross-case aggregation analysis is performed on structured interface anomaly information to obtain various data feature rules. Cross-case aggregation analysis refers to the process of global data mining and pattern summarization of multiple structured interface anomaly information. Its core purpose is to identify common patterns from scattered UI layout anomaly cases, rather than processing individual cases in isolation. The cross-case aggregation analysis process can use statistical, clustering, or machine learning algorithms to aggregate cases with similar anomaly behaviors, the same field types, and related anomaly cause information into abstract rules, thereby extracting reusable data feature rules and providing a knowledge foundation for the generation of subsequent test data. Cross-case aggregation analysis transforms fragmented structured interface anomaly information into common knowledge information, avoiding repetitive work and improving the generalization ability of the problem feature library. For example, when 100 UI layout anomaly cases from different user interfaces all cause layout overflow due to excessive text length, aggregation analysis can extract a general data feature rule, rather than retaining 100 redundant records as data feature rules.

[0040] Optionally, the specific process of cross-case aggregation analysis may include: data collection and key attribute extraction, multidimensional feature clustering, and commonality summarization and rule generation. Specifically: Data collection and key attribute extraction include: gathering all structured interface anomaly information and extracting three core analytical dimensions from the substructure of each structured interface anomaly message: anomaly field data (raw data values), anomaly cause information (technical root cause description), and anomaly problem description information (phenomenon description). Simultaneously, the field type mapped to the anomaly field name corresponding to each structured interface anomaly message is recorded.

[0041] Multidimensional feature clustering: Using natural language processing (NLP) and pattern recognition techniques, the above three dimensions are comprehensively analyzed, including: semantic vectorization and similarity calculation of abnormal cause information and abnormal problem description information; grouping structured interface abnormal information describing similar technical causes and visual phenomena together as case groups to obtain multiple case groups; performing pattern mining on abnormal field data to analyze whether the abnormal data in the clustered structured interface abnormal information have common features (e.g., all exceed a certain length, all contain specific characters, all are in a certain format).

[0042] Commonality Induction and Rule Generation: For each clustered case group, the system can summarize the common features mined from the above patterns to define a new data feature rule, including: summarizing the abnormal field data with common features in the case group to obtain test data examples; summarizing the abnormal cause information with common features in the case group to obtain the common essence of the abnormal cause of the case group, and summarizing the common essence of the abnormal cause of the case group to obtain the abnormal cause information that triggers the user interface; summarizing the abnormal problem description information with common features in the case group to obtain the common phenomenon of the problem manifestation in the case group, and summarizing the common phenomenon of the problem manifestation in the case group to obtain the abnormal problem description information that triggers the user interface. The test data examples, the abnormal cause information that triggers the user interface, and the abnormal problem description information are integrated to obtain the data feature rule.

[0043] For example, based on the aforementioned "product review input box" case (Case A), suppose the system analyzes two more structured cases. Case B: On the "Post Updates" page of a social application, a user enters a long string of text without spaces in the content input box, causing the "Send" button below to be obscured. Case C: On the "Edit Notes" page of a note-taking application, a user pastes a long block of code without line breaks, making the "Save" option at the bottom inaccessible. Through cross-case aggregation analysis, it was identified that the anomaly cause information and problem description information of these three cases were highly similar in semantics (all involving "long text", "height calculation", and "bottom button unusable"). Moreover, their abnormal field data all had the characteristics of "extremely long, unbroken strings". Clustering was performed to generate a data feature rule: Test data example: continuous strings with a length of more than 80 characters and no spaces or line breaks; Triggering user interface anomaly cause information: In scrollable or multi-line text editing areas, inputting extremely long, unbroken text can easily cause the layout engine to seriously miscalculate the total height of the content, destroying the layout constraint relationship between the container and internal and external elements; Anomaly problem description information: The bottom operation area (including buttons, tabs, etc.) below the text input area is squeezed out, covered, or becomes uninterrupted.

[0044] Furthermore, the data feature rules generated from cross-case aggregation analysis are systematically grouped according to the type of the abnormal field. Each group forms an independent data feature pattern, and the set of all data feature patterns constitutes the problem feature library. For example, multiple data feature rules applicable to text input boxes (field type "string") are grouped into the same data feature pattern, while multiple data feature rules applicable to monetary input boxes (field type "numeric") are grouped into another data feature pattern. Each data feature pattern contains at least one data feature rule, and the data feature pattern itself can be appended with metadata (such as a list of applicable UI layout control types, which is not limited). The problem feature library, as a knowledge hub, realizes the transformation from scattered UI anomaly cases to a structured and evolvable problem feature knowledge base, enabling the continuous reuse of information leading to abnormal UI layouts. This significantly improves the coverage of subsequent target test data generation and UI automated testing, and supports dynamic expansion. When a new UI layout anomaly case is added and new data feature rules are generated after aggregation analysis, it is automatically grouped into the data feature pattern of the corresponding field type, realizing continuous optimization of the problem feature library.

[0045] like Figure 1bThe diagram illustrates an example of the internal implementation structure of a data generation model. In this example, the data generation model includes a generative adversarial network (GAN) and a constraint generator. Based on the internal implementation structure of the data generation model, a specific implementation method is provided that inputs data feature rules from the target data feature pattern, the field to be tested, and its constraint rules into the data generation model. Using the constraint rules as boundary conditions, and based on test data examples in the data feature rules, along with their triggering user interface anomaly cause information and anomaly description information, the model generates target test data for the field to be tested. This includes: inputting the data feature rules, the field to be tested, and its constraint rules into the data generation model; within the data generation model: using the generative adversarial network to perform feature parsing and context integration on the test data examples, triggering user interface anomaly cause information, and anomaly description information to obtain a generation strategy; defining the data generation range for the field to be tested using the generation strategy; and using the constraint generator, with the constraint rules as boundary conditions, combined with the generation strategy, generating target test data for the field to be tested.

[0046] In this embodiment, the Generative Adversarial Network (GAN) is a core component of the data generation model responsible for intelligent reasoning and strategy formulation. It is a neural network model inspired by the Generative Adversarial Network architecture, containing a generator and a discriminator. Through continuous adversarial training and collaboration, they learn how to generate test data generation strategies that accurately hit the data feature rules in the target data feature patterns. The GAN in this embodiment is specifically designed and trained to process semantic information in the software testing domain (such as anomaly cause information and anomaly problem information), and its output is a generation strategy that guides data generation.

[0047] Generative Adversarial Networks (GANs) include not only generators and discriminators, but also embedding layers and encoders. The embedding layer is the semantic representation module in a GAN, used to convert information input into the data generation model into continuous vector representations. In this embodiment, the embedding layer is mainly used to vectorize test data examples, information triggering user interface anomalies, and anomaly descriptions. For example, if the anomaly cause information and anomaly descriptions exist in text form, the embedding layer can map the field types, anomaly cause information, and anomaly descriptions in the text into semantic vectors. Through the embedding layer, input information from different sources and in different forms can be uniformly mapped to the same vector space.

[0048] The encoder is a feature abstraction and context modeling module in a generative adversarial network (GAN), used to further extract features and integrate context from the vector representation output by the embedding layer. In this embodiment, the encoder can employ a multi-layer neural network structure, such as a recurrent neural network, a Transformer encoding structure, or a multi-layer fully connected network, to jointly encode the embedded anomaly cause information, anomaly description information, and test data examples. The encoder extracts high-level, discriminative semantic features by performing nonlinear transformations and context modeling on the input vector. For example, the encoder can learn the correlation between anomaly cause information and anomaly description, as well as the common patterns exhibited by different test data examples when triggering UI layout anomalies. Through the feature representation output by the encoder, the generator and discriminator can more accurately understand the data feature rules, thereby generating more accurate target test data and its generation strategy.

[0049] Optionally, an adversarial generative network is used to perform feature parsing and context integration on test data examples, information on the reasons for triggering user interface anomalies, and descriptions of anomalies, to obtain the following specific implementation of the generation strategy: (1) The Generative Adversarial Network (GAN) first receives input from the data feature rules, namely test data examples, information on the reasons for triggering user interface anomalies, and descriptions of the anomalies. Through the embedding layer and encoder (such as Transformer or LSTM) in the GAN, the information in the data feature rules is fused and encoded into a unified, high-dimensional semantic feature vector. This semantic feature vector not only contains the surface features of the anomaly field data, but also deeply integrates the contextual logic of "why" (reason) and "how" (problem description).

[0050] (2) The generator in the adversarial generative network takes the above semantic feature vector as conditional input and generates a generation strategy. The generation strategy can be a set of parameters or a vector in the latent space, which defines the direction of test data generation. For example, the content of the generation strategy can be: within the interval of [minimum length, maximum length], prioritize maximizing the length; in terms of character selection, significantly increase the probability of selecting full-width characters and specific special symbols (such as "—"); in terms of structure, avoid the appearance of spaces or newlines.

[0051] The discriminator in a generative adversarial network evaluates the generation strategy based on knowledge learned from a problem feature library. Its goal is to determine the effectiveness of the generation strategy—that is, whether it can guide the generation of test data that matches the essence of the input anomalies and has a high probability of triggering similar layout problems. The discriminator can output a score representing the effectiveness of the generation strategy.

[0052] In this embodiment of the application, the discriminator in the generative adversarial network is no longer just an auxiliary module in the training phase of the traditional generative adversarial network, but is designed as a core component that continuously participates in decision-making during the inference phase. Based on the abnormal knowledge learned from the problem feature library, it performs real-time discrimination on the generation strategy output by the generator.

[0053] Traditional generative adversarial networks (GANs) typically follow a pattern of "generator and discriminator adversarial during training, and only the generator remaining during inference," with the discriminator no longer participating in the decision-making process during inference. However, this traditional pattern is not applicable in the embodiments of this application. The embodiments of this application explicitly require that, during the inference phase, each generated strategy must be evaluated and scored in real-time by the discriminator. Only generation strategies that pass the discriminator's effectiveness evaluation can be used to constrain the generator to generate the final target test data.

[0054] Specifically, the discriminator, based on data feature patterns and rules in the problem feature library, judges the generation strategy output by the generator and outputs a score indicating the effectiveness of the generation strategy. This score is used to measure the degree of matching between the generation strategy and the anomalous features of the target test data.

[0055] (3) During the training phase, the generator can continuously try different generation strategies to get the discriminator to output a high score, while the discriminator continuously improves its discrimination ability. Through this adversarial game, the network is optimized. During the inference (use) phase, the adversarial generative network with fixed parameters performs forward computation on the input, and the generator outputs the optimized generation strategy.

[0056] Specifically, in this embodiment, the training method for the Generative Adversarial Network (GAN) is specially designed to improve the real-time discrimination capability of the discriminator. The sample data used for training is no longer the "real data / fake data" adversarial samples found in traditional GANs, but instead includes: sample test data, the labeled generation strategy information corresponding to the sample test data, and the labeled discrimination results for that generation strategy. The labeled generation strategy information guides the generator to learn "what generation strategies are correct," while the labeled discrimination results supervise the discriminator to learn "how to determine whether a strategy is effective."

[0057] In the specific training process, sample test data is input into the generator to generate intermediate policy information. This intermediate policy information is then compared with the labeled generated policy information to calculate the generation policy loss function, which is used to constrain the generator's policy generation capability. The intermediate policy information output by the generator is input into the discriminator to obtain intermediate discrimination results. These intermediate discrimination results are then compared with the labeled discrimination results to calculate the discrimination loss function, which is used to improve the discriminator's real-time discrimination accuracy. The weighted sum of the generation policy loss function and the discrimination loss function is used as the overall loss function for training the data generation model.

[0058] Through the above training and inference mechanism, the overall performance of the data generation model is considered to be up to standard only when the generator meets the preset requirements in both the generation accuracy of the generation strategy and the discriminator meets the policy discrimination accuracy. This enables the data generation model to continuously output highly accurate and effective generation strategies during the inference phase, fundamentally avoiding the generation of a large amount of invalid test data and significantly improving the efficiency of UI layout testing.

[0059] The technical effects of Generative Adversarial Networks (GANs) in this embodiment are significantly different from traditional methods. Traditional test data generation relies on preset rules or random sampling, which cannot adapt to the heterogeneity of UI controls, resulting in a large number of invalid tests. In this embodiment, however, the adversarial mechanism dynamically reconstructs the generation strategy each time it is executed. That is, the generator, under the real-time feedback of the discriminator, combines the constraint rules of the current field to be tested with the corresponding target data feature patterns to generate a highly targeted range. This "one-time generation, one-time accuracy" characteristic significantly reduces the number of test rounds, improves UI testing efficiency, and fundamentally solves the problem in traditional methods where massive test cases only cover limited test data.

[0060] Furthermore, a constraint generator is used, with constraint rules as boundary conditions, combined with a generation strategy, to generate target test data for the field to be tested. The constraint generator is the core component of the data generation model responsible for ensuring the legality and compliance of the output test data. It is a specially trained, differentiable function or neural network module. Its core responsibility is to receive the generation strategy from the generative adversarial network and the constraint rules from the UI layout template, and to fuse and compromise between the two, ultimately outputting target test data that fully conforms to business and technical constraints while maximizing the representation of abnormal characteristics.

[0061] Optionally, the following is a specific implementation method for generating target test data for the field to be tested using a constraint generator, with constraint rules as boundary conditions, and in conjunction with a generation strategy: (1) The constraint generator parses the abstract generation strategy into a series of specific and quantifiable data generation operation instructions. At the same time, it encodes the constraint rules into mathematical boundary conditions.

[0062] (2) The generation process is modeled as an optimization problem within the solution space defined by the constraint rules. The constraint generator generates a candidate set of test data guided by the generation strategy. For each candidate test data, the strategy compliance score and constraint violation score are calculated, i.e., the score for conforming to the intention of the generation strategy and the score for violating the constraint rules. The goal of the constraint generator is to maximize the strategy compliance score while ensuring that the constraint violation score is zero.

[0063] (3) Based on the policy compliance and constraint violation, iterative optimization is performed. The constraint generator outputs data that compromises the generation policy under the given constraints, i.e., the target test data.

[0064] For example, when the generation strategy suggests a character length of [18, 22], and the constraint rule limits max_length to 20, the constraint generator dynamically compresses the range to [18, 20] and generates "super_long_username_123" (20 characters) as the target test data.

[0065] The constraint generator can be trained as a standalone model, aiming to master the basic ability to generate compliant data based on arbitrary constraint rules. The training process includes two phases. Specifically: Phase 1: Pre-training Phase. Training data can utilize data feature rules and their corresponding constraint rules from a large problem feature library as training samples. The training objective is to train the constraint generator to become a "conditional data generator." Given a set of constraint rules as input, the neural network model needs to output data that fully satisfies these constraints. The loss function can be designed as a combination of data reconstruction loss and constraint satisfaction loss. The data reconstruction loss measures the difference between the test data generated by the neural network model and the real compliant data in the training samples, such as cross-entropy loss; the constraint satisfaction loss is achieved by using a differentiable "constraint checker" module to check the generated test data and remove any tests that violate the constraint rules. After completing this phase, the constraint generator can generate technically valid test data for any given constraint rule.

[0066] Phase Two: End-to-End Collaborative Training Phase. This phase connects the constraint generator with the Generative Adversarial Network (GAN), including its internal generator and discriminator, for joint training. The goal is to enable the entire data generation model to learn to generate test data that conforms to the generation strategy while adhering to the constraint rules. Training Data: "Abnormal data" validated in UI layout anomaly cases are used as positive samples. This includes field data that satisfies all field constraint rules but actually triggers UI layout anomalies, along with their corresponding constraint rules, anomaly cause information, and anomaly description information.

[0067] The training process (one iteration cycle) includes: Step A (Policy Generation): The generator in the Generative Adversarial Network receives "anomaly cause information" and "problem description information" from the training samples and generates an abstract generation policy.

[0068] Step B (Data Synthesis): The constraint generator receives the generation strategy and the constraint rules in the training samples, and synthesizes a data candidate.

[0069] Step C (Adversarial Evaluation and Optimization): The discriminator in the adversarial generative network evaluates the data candidate, determining its similarity to historical real "compliant anomaly data" (positive samples), i.e., whether it "looks like" a valid field data that triggers an anomaly. Simultaneously, the constraint generator re-verifies whether it meets the boundary conditions.

[0070] Step D (Loss Calculation and Backpropagation): Calculate the triple loss: 1. Calculate policy compliance; 2. Calculate constraint violation; 3. Policy-data alignment loss: Ensure that the test data output by the constraint generator truly reflects the intent of the generation policy. By calculating the above triple loss, the adversarial generative network in the data generation model can learn to extract executable generation policies that accurately target the core of data feature rules. The constraint generator learns to use the constraint rules as boundary conditions and generate target test data along the direction indicated by the generation policy, thereby improving the "effectiveness" and "reliability" of test data in automated testing.

[0071] like Figure 1b As shown, after loading UI layout data through a UI automation tool and performing automated testing on the user interface, the process also includes: capturing an image of the user interface, which is obtained by the UI automation tool loading UI layout data and driving the application to render the UI layout data; referring to step ⑦, inputting the image of the user interface and UI layout data into the multimodal analysis model, and the multimodal analysis model performing step ⑧ to perform visual-field analysis to obtain visual-field analysis results; and generating a test report based on the visual-field analysis results and the problem feature library.

[0072] In this system, the UI automation tool acts as the test execution engine, receiving the instantiated UI layout data generated in the previous step. The UI automation tool can be injected or configured into the application under test (the application containing the user interface) as a set of instructions or scripts. After receiving the UI layout data, the application's rendering engine constructs a complete view tree in memory based on the UI layout control types, positions, sizes, and specific content defined in the UI layout data, and performs pixel-level rendering calculations to draw a concrete, visual user interface on the screen (or virtual display device). The application's rendering engine can be Android's View system, iOS's UIKit, or a web browser's rendering engine; this embodiment does not limit this. By simulating the normal rendering behavior of the application after receiving the target test data through the automation tool, a target user interface that is highly consistent with what the actual user sees is generated for testing and verification, thereby achieving automated testing of the user interface.

[0073] After the user interface is automated, the operating system of the application under test calls the screenshot API (Application Programming Interface) to automatically capture images of the user interface.

[0074] In this embodiment, the multimodal analysis model refers to a large model capable of jointly processing and comprehensively analyzing data from different information modalities. It simultaneously receives image data from the user interface and corresponding UI layout data. The image data of the user interface belongs to visual modal information, reflecting the actual display effect of the user interface after rendering within the application. The UI layout data belongs to structured field modal information, reflecting the logical definitions, layout attributes, and field data of each UI control and its fields. By collaboratively analyzing the aforementioned different modal data—i.e., visual-field analysis—the multimodal analysis model achieves accurate identification and location of abnormal user interface layouts, thereby avoiding the analytical limitations imposed by relying solely on single visual information or single structured data.

[0075] Visual-field analysis refers to the process of jointly analyzing the user interface based on the correspondence between visual information and field information within the multimodal analysis model. The visual-field analysis result is the comprehensive analysis result output by the multimodal analysis model after completing the visual-field analysis. This result simultaneously reflects both the "display results" and "root causes" of anomalies, thus providing structured input for subsequent test report generation. The test report is the test output generated based on the visual-field analysis results and combined with a problem feature library, used to summarize and present the results of automated user interface testing.

[0076] like Figure 1b The diagram illustrates an example of the internal implementation structure of a multimodal analysis model. In this example, the multimodal analysis model includes a layout visual analyzer and a field analyzer. Based on the internal implementation structure of the multimodal analysis model, a specific implementation method for inputting user interface images and UI layout data into the multimodal analysis model for visual-field analysis to obtain visual-field analysis results includes: inputting user interface images and UI layout data into the multimodal analysis model; within the multimodal analysis model: using the layout visual analyzer to perform visual analysis on the user interface image; if an abnormal UI layout area is identified in the user interface image, the abnormal UI control within the abnormal UI layout area is determined, and abnormal description information of the abnormal UI layout area is generated; based on the abnormal UI control, the field analyzer is used to perform field analysis on the UI layout data to determine the abnormal field in the UI layout data corresponding to the abnormal UI control, and the abnormal UI control and its corresponding abnormal field are associated to locate the field data that triggers the abnormality; the abnormal UI layout area, abnormal UI control, abnormal description information, and field data that triggers the abnormality are used as the visual-field analysis results.

[0077] The layout visual analyzer, a visual analysis module within the multimodal analysis model, is used to identify visual anomalies in the rendered user interface image. Based on computer vision algorithms, deep learning visual models, or a combination of both, this analyzer examines the outlines, relative positions, size proportions, and hierarchical relationships of controls in the user interface image to identify anomalies that do not conform to expected layout rules or conventional display patterns. Through the layout visual analyzer, potential UI layout anomalies can be identified directly from the visual effects of the interface without relying on specific field values.

[0078] UI layout anomaly areas refer to localized regions in the user interface image that deviate from the normal or expected interface layout due to abnormal control layout or content display. For example, UI layout anomaly areas may manifest as overlapping UI layout controls, UI layout control content exceeding its display boundaries, UI layout controls being obscured, UI layout controls being misaligned, or controls being incompletely displayed. The layout visual analyzer analyzes the visual characteristics of various regions in the user interface image; if it identifies regions exhibiting the above-mentioned abnormal visual characteristics, it determines that region as a UI layout anomaly area.

[0079] After identifying an abnormal UI layout area, the layout visual analyzer can further perform fine-grained analysis of the visual elements within that area to determine the specific UI layout controls causing the anomaly. Specifically, the layout visual analyzer can distinguish different UI layout controls within the abnormal area through methods such as UI layout control boundary detection, UI layout control outline segmentation, UI layout control hierarchy inference, and spatial relationship analysis between UI layout controls. Based on the overlapping, occlusion, or display integrity of the UI layout controls, it can identify one or more abnormal UI layout controls. The identified abnormal UI layout controls are those that directly affect the layout anomaly or bear the primary responsibility for the abnormal behavior within that area.

[0080] The anomaly description information is a semantic description of the specific manifestation of UI layout anomalies. It is generated by the layout visual analyzer after identifying the abnormal UI layout controls. The anomaly description information can be generated based on the visual characteristics of the abnormal area, the display state of the abnormal UI layout controls, and the spatial relationships between UI layout controls. It describes the type and manifestation of the user interface layout anomaly. For example, the anomaly description information may include "text control content overflows the display area," "image control obscures the button below," "input box width is insufficient, causing content to be cropped," or "abnormal list item height causes layout misalignment," etc. Through the anomaly description information, the visual manifestation of UI layout anomalies in the user interface can be intuitively reflected, providing clear semantic basis for subsequent field analysis and test report generation.

[0081] For example, suppose we perform visual analysis on a test screenshot of a product listing page. The visual analyzer detects a visually anomalous area at the location of the second product item. The bounding box of this area covers the right half of the product title text and the "Favorite" icon to the right. The analyzer determines that the "Product Title Text" control within this anomalous area is the primary element causing the anomaly, as it should be confined to the area to the left of the icon, but its visual content extends beyond that area. Furthermore, the analyzer outputs the following structured anomaly description: Anomaly Type: Text content overflow and overlap; Involved Controls: Product Title Text Tag (ID: item_title_2); Anomaly Measurement: Text overflows to the right, causing approximately 30% overlap with the adjacent "Favorite" icon. Through this process, the layout visual analyzer completes the crucial transformation from the original image to structured anomaly description information, providing precise visual evidence and location data for subsequent association with specific field data.

[0082] Furthermore, based on the abnormal UI control, a field analyzer can be used to perform field analysis on the UI layout data to determine the abnormal field in the UI layout data that corresponds to the abnormal UI control, and associate the abnormal UI control and its corresponding abnormal field to locate the field data that triggered the exception.

[0083] The field analyzer, a field-side analysis module within the multimodal analysis model, parses, retrieves, and locates structured UI layout data to identify field information corresponding to visual anomalies. The field analyzer's role is to further trace the structural definition of the UI layout data, after the layout visual analyzer has already output the abnormal UI layout control, to find the fields bound to that abnormal UI layout control, thereby determining the "data trigger source" of the anomaly—the field data. Through the field analyzer, a mapping from "interface display anomaly" to "field-level data cause" can be achieved, improving the interpretability and traceability of anomaly localization.

[0084] Upon receiving an abnormal UI layout control, the field analyzer determines its corresponding field based on the control's location information within the UI layout data. Specifically, this can be achieved by: extracting the control identifier information of the abnormal UI layout control, such as its control ID, control type, control hierarchy path, or node position in the view tree; retrieving UI layout control nodes matching the control's identifier from the UI layout data; and, upon finding a matching UI layout control node, reading the field binding information recorded within that node to determine the set of fields corresponding to the abnormal UI layout control. If the UI layout control corresponds to multiple fields (e.g., composite controls, list item controls, dynamic form controls), the field analyzer can further filter the field set by combining field constraint rules, field types, and exception description information to determine the abnormal field that triggered the exception.

[0085] To ensure reproducibility and interpretability of anomaly localization, the field analyzer establishes explicit associations between abnormal UI layout controls and their corresponding abnormal fields. In practice, one or more association records can be generated. Each record may include: an abnormal UI layout control identifier (control ID / path / type), an abnormal field identifier (field name / field ID / field type), field binding relationships (e.g., "control display content is bound to field X," "control size constraints are affected by the length of field X"), and field constraint rule information. These association records clearly indicate "which UI layout control's anomaly is related to which field," providing an index for subsequently locating the field data that triggered the anomaly.

[0086] After establishing the association between UI layout controls and their corresponding exception fields, the field analyzer can further read the field data corresponding to the exception fields to determine the specific value that triggered the exception. In practice, the field analyzer can extract the field value, field length, field format characteristics, and their conformity to constraint rules from the exception fields in the UI layout data, and combine this with the exception description information to determine the causal relationship between the field data and the exception. For example, when the exception description information is "text overflow," the field analyzer can compare the field value length with the control's displayable length or layout constraints to determine whether the field value exceeds the display capacity. When the exception description information is "occlusion / overlap," the field analyzer can combine the control size, layout constraints, and field content size estimation (e.g., text rendering width estimation) to infer whether the field value caused the control to expand and trigger overlap. The field analyzer outputs the field data that triggered the exception, that is, the field data in the current UI layout data that is associated with the exception UI layout control and meets the exception triggering characteristics.

[0087] For example, the abnormal UI layout control is a "product title text control." The layout visual analyzer identifies an abnormal description message: "Text overflows and obscures the price control." The field analyzer locates the corresponding node in the UI layout data based on the control ID of the abnormal UI layout control, finds that its bound field is "title," and reads the field data as a long string. The field analyzer further combines field constraint rules (such as no maximum length limit or excessively wide limit) and UI layout control width constraints to determine that the length of the field value does not match the display capability of the UI layout control. Therefore, the "current value of the title field" is identified as the field data that triggered the abnormality. The result of the visual-field analysis is: "In the product list page, the area where the product title text control (item_title_2) is located is identified as an abnormal UI layout area, and its abnormal manifestation is that the text content overflows and obscures the price control; further analysis determines that this abnormality is triggered by the current long string value of the title field." That is, the abnormal area + abnormal UI layout control + abnormal description message + field data that triggered the abnormality together constitute the visual-field analysis result in this example.

[0088] Optionally, the training process for training a multimodal analysis model includes: (1) The training data for the multimodal analysis model can be derived from the recorded UI layout anomaly case library and the normal rendering case library. The normal rendering case library refers to the set of interface samples used to describe and store the user interface under the condition that it conforms to the design specifications and layout expectations. It is used to characterize the standard display state of the UI when there is no layout anomaly. Each case in the normal rendering case library corresponds to an interface rendering result without UI layout anomaly, which can reflect the normal interface performance of the application under the condition of legal field data and reasonable layout constraints. Each training sample contains at least: a user interface image and UI layout data corresponding to the image. For anomaly samples, it also contains anomaly annotation information, such as: UI layout anomaly area annotation (boundary box / segmentation mask), anomaly UI layout control annotation (control ID or control path), anomaly type label and anomaly description information, and field annotation that triggered the anomaly (field name / field ID / field value). By constructing a training sample set aligned with "image-layout data-anomaly annotation" in the above way, a data foundation is provided for subsequent modular training.

[0089] (2) The training process of the layout visual analyzer includes: the layout visual analyzer is trained on user interface images, and its training objective is to identify abnormal UI layout areas, locate abnormal UI layout controls, and output abnormal type or abnormal description information. In specific training, the UI layout abnormal area annotation can be used as a supervision signal, and the abnormal area is learned by object detection or instance segmentation; at the same time, abnormal control annotation is introduced, so that the model can further learn the control-level attribution ability in the UI layout abnormal area, that is, to identify one or more abnormal UI layout controls that cause the abnormality. For the generation or output of abnormal description information, two methods can be used: first, the abnormal type is used as a classification label to train the abnormal type classification head; second, the abnormal description text is used as a supervision signal to train the description generation head, so that the model can output structured abnormal description information based on the visual features of the UI layout abnormal area and the relationship between the controls.

[0090] (3) The training process of the field analyzer includes: the field analyzer is trained on UI layout data, and its training objective is to determine the corresponding abnormal field and locate the field data that triggers the abnormality when the abnormal UI layout control is known. In specific training, the abnormal UI layout control identifier is used as a condition input and is input into the field analyzer together with the UI layout data; the abnormal field label is used as a supervision signal to train the field analyzer to select or sort the abnormal field in the control-bound field set; further, the field data label that triggers the abnormality is used as a supervision signal to enable the field analyzer to locate the specific value or value fragment (such as excessively long text, special symbol sequence, abnormal format value, etc.) that causes the abnormality in the candidate field values.

[0091] (4) Alignment Training Between Visual and Field Sides: To ensure the multimodal analysis model has a stable ability to correspond visual information with field information during actual testing, an alignment training mechanism between the visual and field sides is introduced during the training phase. Specifically, abnormal UI layout controls are used as the association point between the visual and field sides, and consistency constraints are applied to the abnormal UI layout control location results output by the layout visual analyzer and the UI layout control nodes located in the UI layout data by the field analyzer. When the layout visual analyzer identifies and outputs abnormal UI layout controls, the field analyzer retrieves the same UI layout control node in the corresponding UI layout data and outputs the field associated with that UI layout control. By introducing a consistency loss function, constraints can be applied to the UI layout control location results on the visual side and the UI layout control retrieval results on the field side, ensuring consistency between the two at the UI layout control level and the field selection level. This improves the overall stability of locating abnormal UI layout controls from the user interface image, locating abnormal fields from the UI layout controls, and then locating the data that triggers abnormal fields.

[0092] Optionally, to improve the model's ability to identify complex UI layout anomalies, a hard example mining mechanism can be further introduced during training. Specifically, false positives, false negatives, samples with unclear anomaly boundaries, and samples that are difficult to distinguish between different anomaly types generated during the automated testing phase are re-included into the training dataset as hard example samples. Simultaneously, by combining high-risk data feature rules summarized in the problem feature library, such as excessively long text, extreme values, special character combinations, and abnormal rich text structures, more anomaly triggering samples are automatically constructed to expand the distribution of anomaly samples.

[0093] By periodically training the aforementioned difficult and newly added samples, the multimodal analysis model can be continuously updated and optimized as UI layout anomaly cases accumulate, thereby maintaining generalization ability and robustness in application scenarios where user interface structure changes frequently.

[0094] After training, the layout visual analyzer can output abnormal UI layout areas, abnormal UI layout controls, and abnormal description information from the user interface image during the inference phase; the field analyzer can locate abnormal fields in the UI layout data based on the abnormal UI layout controls during the inference phase and provide the field data that triggered the abnormality; the outputs of both are uniformly encapsulated into visual-field analysis results, which are further used to generate test reports in conjunction with the problem feature library, thereby realizing an automated closed loop from screenshot to field.

[0095] In one optional embodiment, a test report is generated based on the visual-field analysis results and in conjunction with the problem feature library, including: determining the data feature rules corresponding to the field data that triggered the anomaly based on the visual-field analysis results and in conjunction with the problem feature library; and generating a test report based on the UI layout anomaly area, the abnormal UI control, the field data that triggered the anomaly and its field type, and the corresponding data feature rules.

[0096] Based on the visual-field analysis results and combined with the problem feature library, the data feature rules corresponding to the field data that triggers the anomaly are determined. A specific UI layout anomaly is then associated and attributed to the data feature patterns in the problem feature library. By mapping the field data that triggers the anomaly to the data feature rules in the problem feature library, it can be clarified whether the anomaly belongs to a known anomaly type, whether it conforms to a certain high-risk data feature pattern, and the common characteristics of the anomaly in historical UI layout anomaly cases. This avoids merely remaining at the level of "discovering anomalies" but further reveals the underlying data feature patterns, thereby improving the systematic nature and reusability of anomaly analysis.

[0097] By identifying the data feature rules corresponding to the fields that trigger anomalies, standardized descriptions and categorized management of UI layout anomalies can be achieved. On one hand, this helps to uniformly classify anomalies triggered by the same data features across different user pages and UI layout controls, reducing the complexity of problem analysis. On the other hand, anomalies discovered in current testing can be compared with existing data feature rules in the problem feature library to quickly determine whether an anomaly is a duplicate, known, or high-risk issue, thereby improving the efficiency of users in judging the severity and priority of anomalies. Furthermore, this mapping result can be used in reverse to optimize test data generation strategies, improving the coverage of similar anomalies in subsequent automated tests.

[0098] When generating test reports, the results of visual-field analysis and data feature rules from the problem feature library can be used comprehensively. Specifically, the generation of test reports can be completed directly by a multimodal analysis model or by a large model with natural language generation capabilities (such as a general-purpose large language model or a domain-customized language model). This application embodiment does not limit this. By introducing a large model, structured anomaly analysis results can be transformed into more readable and semantically complete test report text.

[0099] The test report can include the following: the identifier of the user page or interface where the anomaly occurred, the location information of the abnormal UI layout area, the abnormal UI layout controls involved, the visual manifestation of the anomaly and the corresponding anomaly description, the field name and field type that triggered the anomaly, the specific field data that triggered the anomaly, the matched data feature rules and their descriptions, the anomaly cause information, and the anomaly problem description information. By presenting the visual anomaly information, the data cause at the field level, and the historical knowledge at the data feature rule level in a unified manner, the test report can comprehensively reflect the occurrence mechanism and scope of impact of UI layout anomalies.

[0100] The test report generated using the above method not only visually displays layout anomalies occurring in the user interface, but also clearly explains which field data triggered the anomaly, what data characteristic rules it conforms to, and the anomaly's location in historical issues, thus significantly improving the understandability and operability of the anomaly analysis results. This test report can directly serve as the basis for development fixes, regression testing, and issue review, helping to reduce the testing cost of UI layout issues.

[0101] In one optional embodiment, based on the visual-field analysis results and in conjunction with the problem feature library, the data feature rules corresponding to the field data that triggered the anomaly are determined, including: matching data feature patterns from the problem feature library according to the field type corresponding to the field data that triggered the anomaly; if the match is successful, matching data feature rules in the matched data feature patterns according to the anomaly description information and the field data that triggered the anomaly; if the match is successful, the matched data feature rules are used as the data feature rules corresponding to the field data that triggered the anomaly; if matching data feature patterns from the problem feature library fails, or matching data feature rules in the data feature patterns fails, the UI layout anomaly area, the anomaly UI layout control, the anomaly description information, and the field data that triggered the anomaly are recorded to obtain user interface anomaly details information, and data feature rules are generated according to the user interface anomaly details information as the data feature rules corresponding to the field data that triggered the anomaly.

[0102] The process involves matching data feature patterns from a problem feature library based on the field type of the data triggering the anomaly. This aims to narrow the matching range and improve the accuracy of anomaly attribution. The problem feature library pre-categorizes data feature rules according to field type, with each field type corresponding to at least one data feature pattern. In practice, the field type corresponding to the triggering anomaly data can be read from the visual-field analysis results, such as text, numeric, or structured object fields. This field type is then used as a search condition to perform a field type-level matching operation in the problem feature library. This matching operation can be achieved through field type identifier comparison, field type enumeration mapping, or field type vector similarity calculation. When a data feature pattern matching the field type exists in the problem feature library, the data feature pattern is considered a successful match.

[0103] After successfully matching a data feature pattern, the system further matches specific data feature rules within that pattern based on the anomaly description and the field data that triggered the anomaly. This distinguishes different anomaly triggering methods within the same field type. For example, even within the same text field, different data feature rules might correspond to "excessive text overflow," "layout anomaly caused by special characters," or "height anomaly caused by line breaks." In the specific implementation, data feature rule matching can be accomplished through one or more of the following methods: First, based on semantic matching of anomaly description information, the anomaly description information is encoded into a semantic vector, and semantic similarity is calculated with the anomaly problem description information stored in the data feature rules; Second, based on field data feature matching, feature indicators (such as character length, character type distribution, format features, etc.) are extracted from the field data that triggers the anomaly, and compared with the test data example features recorded in the data feature rules; Third, logical judgments based on rule constraints, such as determining whether field data meets the threshold for exception triggering conditions defined in a certain rule.

[0104] When the exception description information and the field data simultaneously meet the matching conditions of a certain data feature rule, the data feature rule is determined to be a successful match, and the matched data feature rule is identified as the data feature rule corresponding to the field data that triggered the exception.

[0105] If matching data feature patterns from the issue feature library fails, or if no data feature rule is found among the already matched data feature patterns, it indicates that the current UI layout anomaly belongs to a new type of anomaly or variant anomaly not yet covered by the issue feature library. In this case, the system will enter the anomaly recording and rule generation process.

[0106] Specifically, the multimodal analysis model can uniformly encapsulate and record the abnormal UI layout areas, abnormal UI layout controls, abnormal description information, and field data that triggered the abnormality obtained during the current testing process, forming user interface abnormality details. All of the above information originates from the output of the multimodal analysis model. Specifically, the abnormal image comes from the screenshot step; the abnormal UI layout controls and abnormal description information come from the layout visual analyzer; the abnormal fields and field data come from the field analyzer; and the abnormal cause information is automatically generated through causal analysis between the abnormal description information and the field data. User interface abnormality details include at least the following: abnormal image of the user interface, abnormal UI layout controls and their corresponding abnormal fields, abnormal field data, abnormal cause information, and abnormal problem description information. The abnormal cause information describes the triggering relationship between the field data and the UI layout abnormality, such as "the field value length exceeds the control's display capacity"; the abnormal problem description information provides a semantic summary of the visual manifestation of the abnormality, such as "text overflow causes control overlap." This user interface abnormality details information is automatically generated by the multimodal analysis model during a complete visual-field analysis process and stored in a structured form for subsequent rule abstraction and problem feature library updates.

[0107] After obtaining detailed information about the user interface anomaly, the multimodal analysis model further performs cross-field and cross-case abstract analysis on the anomaly to generate new data feature rules. In specific implementation, the multimodal analysis model can extract common features of the anomaly field data (such as field type, data length range, special format features, etc.) and associate them with the anomaly description information and anomaly cause information to form a reusable data feature rule. The generated data feature rule includes at least: the anomaly image of the user interface, the anomaly UI layout control and its corresponding anomaly field, the anomaly field data, the anomaly cause information, and the anomaly description information. This newly generated data feature rule is identified as the data feature rule corresponding to the field data that triggered the anomaly and can be further stored in the problem feature library.

[0108] For example, in a UI automation test, a multimodal analysis model detected a UI layout anomaly on a product list page: a product title text control was "overflowing and obscuring the price control." The field analyzer determined that the triggering field was "title," a text field containing a very long string of mixed Chinese and English characters. The multimodal analysis model matched the text data feature pattern in the problem feature library based on the "text" field type, and the match was successful. However, based on the anomaly description "text overflowing and obscuring" and the field data features, it attempted to match specific data feature rules within this text data feature pattern, but no existing rule was found. In this case, the multimodal analysis model recorded the abnormal area, the product title text control, the anomaly description, and the current value of the title field as user interface anomaly details, and further analyzed the cause of the anomaly to be "the cumulative character width of the field value exceeds the control layout constraints." Based on this anomaly details, the multimodal analysis model generated a new data feature rule. This data feature rule was determined as the data feature rule corresponding to the triggering field data and stored in the problem feature library for subsequent testing.

[0109] like Figure 1b As shown in step 9, based on the above test report, the test report can be input into the UI layout template optimization model. A repair strategy for user interface anomalies can be generated according to the test report. Based on the repair strategy, the constraint rules of abnormal UI controls in the UI layout template can be adjusted to generate an optimized UI layout template. Further, step 10 is executed to re-perform automated user interface testing on the optimized UI layout template.

[0110] Based on the generated test report, it can be further used as input to introduce a UI layout template optimization model, enabling automated optimization of layout templates with UI layout anomalies. The test report clearly identifies the abnormal UI layout areas, abnormal UI layout controls, anomaly descriptions, triggering field data, and their corresponding data characteristic rules. This information provides accurate and interpretable anomaly localization for subsequent UI layout template optimization. This process achieves an automated closed loop from "anomaly detection" to "template repair" and then to "verifying the repair effect."

[0111] The UI layout template optimization model is a large-scale model used to automatically generate UI layout repair solutions based on test reports. It can infer the unreasonable constraints of the UI layout space in the UI layout template that caused the anomaly based on identified UI layout anomalies and their data triggering characteristics, and output corresponding repair strategies. The UI layout template optimization model can be a rule-based reasoning model, a machine learning model, or a generative model combining a large-scale model; this application embodiment does not limit this. The UI layout template optimization model can perform targeted optimization of UI layout templates without relying on manual analysis, thereby reducing the manual cost of resolving UI layout issues.

[0112] The repair strategy refers to a set of layout adjustment suggestions generated in response to UI layout anomalies. These suggestions guide the modification of constraint rules for abnormal UI layout controls within the UI layout template. A repair strategy can include at least: the identifier of the abnormal UI layout control, the type of constraint rule to be adjusted, the direction or range of adjustment parameters, and the expected effect after adjustment. For example, a repair strategy might instruct "increase the maximum width of the text control," "add automatic line wrapping constraints to the text control," "add minimum spacing constraints to adjacent controls," or "set an upper limit constraint for field data length." The goal of the repair strategy is to eliminate the triggering conditions of the anomaly, ensuring that the UI layout remains stable under various test data inputs.

[0113] Optionally, the generation of remediation strategies can be achieved in various ways. For example, the UI layout template optimization model can look up UI layout anomaly cases or predefined remediation rule mapping relationships in the problem feature library based on the anomaly description information and corresponding data feature rules in the test report, thereby generating targeted remediation strategies; or, by using a large model with reasoning capabilities, the test report can be used as input to automatically generate remediation suggestions that conform to layout design specifications.

[0114] After obtaining the repair strategy, the constraint rules of abnormal UI layout controls in the UI layout template are automatically adjusted according to the repair strategy. Specifically, the UI layout template node corresponding to the abnormal UI layout control can be located within the UI layout template, and its layout constraint parameters can be modified. For example, the width and height constraints of the control can be updated, the alignment adjusted, adaptive or line-wrapping rules added, minimum / maximum size limits modified, or spacing constraints with adjacent controls adjusted. Through this method, local constraint optimization of abnormal UI layout controls can be performed without changing the overall interface structure, thereby generating an optimized UI layout template.

[0115] After generating the optimized UI layout template, reusing it for user interface automated testing and re-executing steps ①-⑥ can verify the effectiveness and stability of the remediation strategy. By reloading the optimized UI layout template and inputting the same or similar test data, the system can automatically determine whether the original UI layout anomalies have been eliminated and whether new layout problems have been introduced. This process forms an automated closed-loop mechanism of "user interface testing—remediation strategy generation—UI layout template optimization—retesting," improving the robustness and adaptability of the UI layout template and significantly increasing the overall efficiency of UI automated testing and user interface optimization.

[0116] It should be noted that in the above embodiments, a large language model refers to a model whose parameters meet the set parameter quantity requirements, and the parameter quantity requirements are not limited. Different definitions may exist in different scenarios and domains. For example, in some scenarios or domains, a large language model refers to a model with a parameter scale reaching tens of billions, hundreds of billions, or even trillions; this definition of the model's parameter quantity is merely an example.

[0117] The detailed implementation methods and beneficial effects of each step in this embodiment have been described in detail in the foregoing embodiments, and will not be elaborated here.

[0118] Furthermore, some processes described in the above embodiments and accompanying drawings include multiple operations appearing in a specific order. However, it should be clearly understood that these operations may not be executed in the order they appear herein, or they may be executed in parallel. The operation numbers, such as 101, 102, etc., are merely used to distinguish different operations and do not represent any execution order. Additionally, these processes may include more or fewer operations, and these operations may be executed sequentially or in parallel. It should be noted that the descriptions such as "first" and "second" in this document are used to distinguish different messages, devices, modules, etc., and do not represent a sequential order, nor do they limit "first" and "second" to different types.

[0119] Figure 2 This is a schematic diagram of an electronic device structure provided for an exemplary embodiment of this application. For example... Figure 2 As shown, the electronic device includes: a memory 24 and a processor 25.

[0120] Memory 24 is used to store computer programs and can be configured to store various other data to support operation on the electronic device. Examples of this data include instructions for any application or method used to operate on the electronic device, UI layout templates, data feature rules, and target test data, etc.

[0121] The memory 24 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0122] Processor 25, coupled to memory 24, executes a computer program in memory 24 for: acquiring a UI layout template of a user interface whose layout has changed, the UI layout template containing UI layout controls and their corresponding fields to be tested and constraint rules; determining the field type of the field to be tested, and matching a target data feature pattern from a problem feature library based on the field type; the problem feature library includes multiple data feature patterns, each data feature pattern including at least one data feature rule, each data feature rule including a test data example and its triggering user interface anomaly information and anomaly description information; inputting the data feature rules in the target data feature pattern, the field to be tested, and its constraint rules into a data generation model, using the constraint rules as boundary conditions, and generating target test data for the field to be tested based on the test data example in the data feature rules and its triggering user interface anomaly information and anomaly description information; backfilling the target test data into the UI layout template to obtain instantiated UI layout data; and loading the UI layout data through a UI automation tool to perform automated testing of the user interface.

[0123] In an optional embodiment, the processor 25 acquires UI layout anomaly cases, which include recorded user interface anomaly details. These details include at least an abnormal image of the user interface, abnormal UI layout controls and their corresponding abnormal fields, abnormal field data, anomaly cause information, and anomaly problem description information. Following the order of UI control positioning, abnormal field association, information extraction, and structured encapsulation, and according to a preset structured field system, the abnormal image, abnormal UI layout controls, abnormal fields, abnormal field data, anomaly cause information, and anomaly problem description information are structured to obtain structured interface anomaly information. The preset structured field system is hierarchical. Cross-case aggregation analysis is performed on the structured interface anomaly information to obtain various data feature rules. Based on the field type of the abnormal fields, these various data feature rules are categorized to obtain a problem feature library including at least one data feature pattern.

[0124] In one optional embodiment, the data generation model includes: a generative adversarial network and a constraint generator; the processor 25 inputs data feature rules from the target data feature pattern, the field to be tested, and its constraint rules into the data generation model, and uses the constraint rules as boundary conditions to generate target test data for the field to be tested based on test data examples and their triggering user interface anomaly cause information and anomaly problem description information in the data feature rules, including: inputting data feature rules, the field to be tested, and its constraint rules into the data generation model; within the data generation model: using the generative adversarial network to perform feature parsing and context integration on the test data examples, triggering user interface anomaly cause information, and anomaly problem description information to obtain a generation strategy; the generation strategy defines the data generation range for the field to be tested; and using the constraint generator, using the constraint rules as boundary conditions, combined with the generation strategy, to generate target test data for the field to be tested.

[0125] In an optional embodiment, after the processor 25 loads UI layout data through a UI automation tool and performs automated testing on the user interface, it further includes: capturing an image of the user interface, which is obtained by the UI automation tool loading UI layout data and driving the application to render the UI layout data; inputting the image of the user interface and the UI layout data into a multimodal analysis model for visual-field analysis to obtain visual-field analysis results; and generating a test report based on the visual-field analysis results and a problem feature library.

[0126] In one optional embodiment, the multimodal analysis model includes: a layout visual analyzer and a field analyzer; the processor 25 inputs the user interface image and UI layout data into the multimodal analysis model for visual-field analysis to obtain visual-field analysis results, including: inputting the user interface image and UI layout data into the multimodal analysis model; within the multimodal analysis model: using the layout visual analyzer to perform visual analysis on the user interface image; if a UI layout abnormality area is identified in the user interface image, the abnormal UI control in the UI layout abnormality area is determined, and abnormal description information of the UI layout abnormality area is generated; based on the abnormal UI control, the field analyzer is used to perform field analysis on the UI layout data to determine the abnormal field in the UI layout data corresponding to the abnormal UI control, and the abnormal UI control and its corresponding abnormal field are associated to locate the field data that triggers the abnormality; the UI layout abnormality area, abnormal UI control, abnormal description information, and field data that triggers the abnormality are used as the visual-field analysis results.

[0127] In one optional embodiment, the processor 25 generates a test report based on the visual-field analysis results and in conjunction with a problem feature library, including: determining the data feature rules corresponding to the field data that triggered the anomaly based on the visual-field analysis results and in conjunction with the problem feature library; and generating a test report based on the UI layout anomaly area, the abnormal UI control, the field data that triggered the anomaly and its field type, and the corresponding data feature rules.

[0128] In one optional embodiment, the processor 25 determines the data feature rules corresponding to the field data that triggers the anomaly based on the visual-field analysis results and in conjunction with the problem feature library. This includes: matching data feature patterns from the problem feature library according to the field type corresponding to the field data that triggers the anomaly; if the match is successful, matching data feature rules in the matched data feature patterns according to the anomaly description information and the field data that triggers the anomaly; if the match is successful, using the matched data feature rules as the data feature rules corresponding to the field data that triggers the anomaly; if matching data feature patterns from the problem feature library fails, or matching data feature rules in the data feature patterns fails, then the UI layout anomaly area, the anomaly UI control, the anomaly description information, and the field data that triggers the anomaly are recorded to obtain user interface anomaly details information, and data feature rules are generated according to the user interface anomaly details information as the data feature rules corresponding to the field data that triggers the anomaly.

[0129] In an optional embodiment, the processor 25 inputs the test report into the UI layout template optimization model, generates a repair strategy for user interface anomalies based on the test report, and adjusts the constraint rules of abnormal UI controls in the UI layout template according to the repair strategy to generate an optimized UI layout template; and re-performs automated user interface testing on the optimized UI layout template.

[0130] Furthermore, such as Figure 2 As shown, the electronic device also includes other components such as a communication component 26, a display 27, a power supply component 28, and an audio component 29. Figure 2 The diagram only shows some components and does not mean that the electronic device includes only these components. Figure 2 The components shown.

[0131] Accordingly, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed, can implement the steps that can be performed by an electronic device in the above method embodiments.

[0132] The above Figure 2The communication component is configured to facilitate wired or wireless communication between the device containing the communication component and other devices. The device containing the communication component can access wireless networks based on communication standards, such as WiFi, 2G, 3G, 4G / LTE, 5G, or combinations thereof. In one exemplary embodiment, the communication component receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, the communication component further includes a Near Field Communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on Radio Frequency Identification (RFID), Infrared Data Association (IrDA) technology, Ultra-Wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0133] The above Figure 2 The display includes a screen, which may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen can be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors can sense not only the boundaries of the touch or swipe action, but also the duration and pressure associated with the touch or swipe operation.

[0134] The above Figure 2 The power supply component provides power to the various components of the device in which it resides. The power supply component may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device in which it resides.

[0135] The above Figure 2 The audio component can be configured to output and / or input audio signals. For example, the audio component includes a microphone (MIC) configured to receive external audio signals when the device containing the audio component is in an operating mode, such as call mode, recording mode, or voice recognition mode. The received audio signals can be further stored in memory or transmitted via a communication component. In some embodiments, the audio component also includes a speaker for outputting audio signals.

[0136] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0137] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in one or more blocks of the flowchart illustrations and / or one or more blocks of the block diagrams.

[0138] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means that implement the functions specified in one or more flowcharts and / or one or more block diagrams.

[0139] These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, such that the instructions, which execute on the computer or other programmable apparatus, provide steps for implementing the functions specified in one or more flowcharts and / or one or more block diagrams.

[0140] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0141] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0142] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0143] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0144] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A method for automated testing of user interfaces, characterized in that, include: Obtain the UI layout template of the user interface whose layout has changed. The UI layout template includes UI layout controls and their corresponding fields to be tested and constraint rules. The field type of the field to be tested is determined, and the target data feature pattern is matched from the problem feature library according to the field type. The problem feature library includes multiple data feature patterns, each data feature pattern includes at least one data feature rule, and each data feature rule includes a test data example and its triggering user interface exception reason information and exception problem description information. The data feature rules in the target data feature pattern, the field to be tested and its constraint rules are input into the data generation model. Using the constraint rules as boundary conditions, target test data is generated for the field to be tested based on the test data examples in the data feature rules and the information on the reasons for triggering user interface anomalies and the description of the anomalies. The target test data is backfilled into the UI layout template to obtain instantiated UI layout data; The UI layout data is loaded using a UI automation tool to perform automated testing on the user interface.

2. The method according to claim 1, characterized in that, Also includes: Obtain UI layout exception cases, which include recorded user interface exception details. The details include at least the exception image of the user interface, the exception UI layout control and its corresponding exception fields, exception field data, exception cause information and exception problem description information. Following the order of UI control positioning, abnormal field association, information extraction, and structured encapsulation, and in accordance with the preset structured field system, the abnormal image, abnormal UI layout control, abnormal field, abnormal field data, abnormal cause information, and abnormal problem description information are structured to obtain structured interface abnormal information; wherein, the preset structured field system is a hierarchical structure. Cross-case aggregation analysis was performed on the structured interface anomaly information to obtain various data feature rules; Based on the field type of the abnormal field, the various data feature rules are categorized to obtain a problem feature library that includes at least one data feature pattern.

3. The method according to claim 1, characterized in that, The data generation model includes: a generative adversarial network and a constraint generator; The data feature rules in the target data feature pattern, the field to be tested, and its constraint rules are input into the data generation model. Using the constraint rules as boundary conditions, target test data is generated for the field to be tested based on the test data examples in the data feature rules, their triggering user interface anomaly cause information, and anomaly problem description information. This includes: The data feature rules, the field to be tested, and its constraint rules are input into the data generation model, and within the data generation model: Using the aforementioned generative adversarial network, feature parsing and context integration are performed on the test data examples, information on the reasons for triggering user interface anomalies, and descriptions of anomalies to obtain a generation strategy; the generation strategy defines the data generation range for the field to be tested. Using the constraint generator, with the constraint rules as boundary conditions, and in conjunction with the generation strategy, target test data is generated for the field to be tested.

4. The method according to claim 1, characterized in that, After loading the UI layout data using a UI automation tool and performing automated testing on the user interface, the process also includes: The image of the user interface is captured. The user interface is obtained by the UI automation tool loading the UI layout data and driving the application to render the UI layout data. The user interface image and UI layout data are input into a multimodal analysis model for visual-field analysis to obtain the visual-field analysis results. Based on the visual-field analysis results and the problem feature library, a test report is generated.

5. The method according to claim 4, characterized in that, The multimodal analysis model includes a layout visual analyzer and a field analyzer. The user interface image and UI layout data are input into the multimodal analysis model for visual-field analysis to obtain visual-field analysis results, including: The image of the user interface and the UI layout data are input into the multimodal analysis model, within which: Using the layout visual analyzer, the image of the user interface is visually analyzed. If an abnormal UI layout area is detected in the image of the user interface, the abnormal UI layout control in the abnormal UI layout area is determined, and abnormal description information of the abnormal UI layout area is generated. Based on the abnormal UI layout control, the field analyzer is used to perform field analysis on the UI layout data to determine the abnormal field in the UI layout data that corresponds to the abnormal UI control, and the abnormal UI control and its corresponding abnormal field are associated to locate the field data that triggered the abnormality. The abnormal UI layout area, the abnormal UI layout control, the abnormal description information, and the field data that triggered the abnormality are used as the visual-field analysis results.

6. The method according to claim 5, characterized in that, Based on the visual-field analysis results and the problem feature library, a test report is generated, including: Based on the visual-field analysis results and the problem feature library, the data feature rules corresponding to the field data that triggered the anomaly are determined. A test report is generated based on the abnormal UI layout area, the abnormal UI layout control, the field data that triggered the abnormality and its field type, and the corresponding data feature rules.

7. The method according to claim 6, characterized in that, Based on the visual-field analysis results and in conjunction with the problem feature library, the data feature rules corresponding to the field data that triggered the anomaly are determined, including: Based on the field type corresponding to the field data that triggered the exception, a data feature pattern is matched from the problem feature library; if the match is successful, a data feature rule is matched in the matched data feature pattern based on the exception description information and the field data that triggered the exception; if the match is successful, the matched data feature rule is used as the data feature rule corresponding to the field data that triggered the exception. If matching the data feature pattern from the problem feature library fails, or matching the data feature rule from the data feature pattern fails, then the abnormal UI layout area, the abnormal UI layout control, the abnormal description information, and the field data that triggered the abnormality are recorded to obtain the user interface abnormality details information. Based on the user interface abnormality details information, data feature rules are generated as the data feature rules corresponding to the field data that triggered the abnormality.

8. The method according to any one of claims 4-7, characterized in that, Also includes: The test report is input into the UI layout template optimization model. Based on the test report, a repair strategy for user interface anomalies is generated. Based on the repair strategy, the constraint rules of abnormal UI controls in the UI layout template are adjusted to generate an optimized UI layout template. The optimized UI layout template was then retested using user interface automation.

9. An electronic device, characterized in that, include: A processor and a memory, the memory storing a computer program that, when executed by the processor, causes the processor to perform the steps of the method as described in any one of claims 1-8.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it causes the processor to perform the steps of the method according to any one of claims 1-8.

11. A computer program product, characterized in that, Includes a computer program / instruction that, when executed by a processor, causes the processor to perform the steps of the method according to any one of claims 1-8.

Images