Automatic testing method and system for AI vision

Abstract

The invention provides an automatic testing method and system for AI vision, and is applied to the field of software UI automatic testing. According to the method, mobile terminal UI automatic test requirements are processed based on a visual processing model and a large language model, a JSON format test case is generated, and a YAML format test case for system execution and an XMIND format test case for test personnel to check and edit are generated after conversion and integrity check; the test equipment information is processed based on Appium service and WebDriver, and equipment interface screenshot information is obtained; processing the equipment interface screenshot and the element description information based on a visual processing model to generate element relative coordinates, and generating an operation result through coordinate conversion and operation processing; performing screenshot verification processing on the operation result to generate a test result record; and processing the test result record, the test case in the YAML format and the test case in the XMIND format to generate test report information.

Description

Technical Field

[0001] This invention relates to the field of automated testing of software UI, and in particular to an automated testing method and system for AI vision. Background Technology

[0002] In the current field of software UI automation testing, mobile UI automation testing is crucial, but traditional testing methods have many shortcomings. Traditional UI automation testing mainly relies on element location (such as XPath, ID, Class, etc.) to identify and manipulate interface elements. In this approach, element location rules are highly susceptible to changes in the interface and can easily become ineffective. For example, when the app interface undergoes a slight layout adjustment, the original XPath-based location rules may fail to accurately locate the target element, leading to test failure. Furthermore, multi-platform adaptation is fraught with difficulties. Due to the differences in interface characteristics and interaction methods between different platforms (such as Android and iOS), different location strategies need to be written for each platform, which greatly increases the complexity and workload of testing.

[0003] From a test script maintenance perspective, the cost is extremely high. Any changes to the interface require extensive modifications to the test code. For example, if a button ID on a page of the app changes, all test code related to that button's operation needs to be modified accordingly, which is not only time-consuming and labor-intensive but also prone to introducing new errors. Furthermore, traditional testing methods have limited support for dynamically generated elements and complex interaction patterns. As app functionality becomes increasingly complex, dynamically generated elements (such as pop-ups and prompts generated based on real-time user actions) are becoming more common, and complex interaction patterns (such as gesture operations and multi-element interactive sequences) are constantly emerging. However, traditional testing methods struggle to effectively handle these situations, failing to comprehensively and accurately test the app, severely impacting testing efficiency and quality. These problems urgently require new technical solutions to improve the level of automated mobile UI testing.

[0004] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this disclosure, and therefore includes information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0005] The purpose of this application is to provide an automated testing method and system for AI vision, which at least to some extent overcomes the problems existing in the prior art. It acquires test requirements, model, and device information, and generates test cases in various formats using vision and large language models. It uses Appium services and WebDriver to obtain device interface screenshots, analyzes the screenshots and element descriptions through a vision processing model, simulates operations to obtain results, verifies the operation results using screenshots, and generates test result records. It generates test reports by integrating various key information, solving problems such as easy failure in traditional test element location and difficulty in multi-platform adaptation. It supports natural language description of test cases and cross-platform testing, effectively improving the stability, efficiency, and coverage of testing.

[0006] Other features and advantages of this application will become apparent from the following detailed description, or may be learned in part by practice of the invention.

[0007] According to one aspect of this application, an automated testing method for AI vision is provided, comprising: acquiring mobile UI automated testing requirements, a visual processing model, a large language model, and test device information; processing the mobile UI automated testing requirements based on the visual processing model and the large language model to generate JSON format test cases, and generating YAML format test cases for system execution and XMIND format test cases for testers to view and edit after conversion and integrity checks; processing the test device information based on Appium service and WebDriver to acquire device interface screenshot information; processing the device interface screenshots and element description information based on the visual processing model to generate element relative coordinates, and generating operation results after coordinate transformation and operation processing; performing screenshot verification processing on the operation results to generate test result records; and processing the test result records and YAML format test cases and XMIND format test cases to generate test report information.

[0008] Another aspect of this application discloses an automated testing device for AI vision, characterized by comprising: an acquisition module for acquiring mobile UI automated testing requirements, a visual processing model, a large language model, and testing device information; a processing module for processing the mobile UI automated testing requirements based on the visual processing model and the large language model, generating JSON format test cases, and after conversion and integrity checks, generating YAML format test cases for system execution and XMIND format test cases for testers to view and edit; processing the testing device information based on Appium service and WebDriver to acquire device interface screenshot information; processing the device interface screenshots and element description information based on the visual processing model to generate element relative coordinates, and after coordinate transformation and operation processing, generating operation results; performing screenshot verification processing on the operation results to generate test result records; and processing the test result records and YAML format test cases and XMIND format test cases to generate test report information.

[0009] According to another aspect of this application, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a second processor, implements the above-described automated testing method for AI vision.

[0010] This application provides an AI vision-based automated testing method and system, with a server designed to improve the efficiency and accuracy of automated mobile UI testing. The system first acquires test requirements, models, and device information, then generates test cases in various formats using visual and large language models. It uses Appium services and WebDriver to obtain screenshots of the device interface, analyzes the screenshots and element descriptions through a visual processing model, and simulates operations to obtain results. The system then verifies the operation results using screenshots and generates test result records. Finally, it integrates various key information to generate a test report. This system solves the problems of easy failure in traditional test element location and difficulty in multi-platform adaptation, supports natural language description of test cases and cross-platform testing, and effectively improves the stability, efficiency, and coverage of testing.

[0011] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0012] Figure 1 A flowchart illustrating an automated testing method for AI vision provided in an embodiment of this application is shown.

[0013] Figure 2 An architecture diagram of an automated testing system for AI vision provided in one embodiment of this application is shown;

[0014] Figure 3A schematic diagram of the structure of an automated testing device for AI vision provided in an embodiment of this application is shown. Detailed Implementation

[0015] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0016] The following is combined with Figure 1 This application describes an automated testing method for AI vision according to exemplary embodiments. In one embodiment, this application also proposes an automated testing method and system for AI vision. Figure 1 As shown, this method is applied to a server and includes:

[0017] S101: Obtain mobile UI automation testing requirements, visual processing model, large language model, and testing equipment information.

[0018] In one implementation, in the AI ​​vision automated testing system, obtaining mobile UI automated testing requirements, visual processing models, large language models, and testing equipment information is the foundation for carrying out testing work. Each part of the information has different characteristics in terms of type, level, structure, and parameters, and they are interconnected, jointly supporting the entire testing process.

[0019] like Figure 2 As shown, mobile UI automation testing requirements include functional requirements, such as interface element interaction functions and business process navigation requirements; and adaptation requirements for different system versions and device models. From macro to micro, they can be divided into system-level requirements, covering the testing scope and objectives of the entire mobile application; module-level requirements, targeting specific testing needs for each functional module; and interface element-level requirements, focusing on the attributes, operations, and display requirements of individual interface elements. These are presented in structured documents or data formats, including fields such as requirement description, operation steps, and operation results. For example, using JSON format, each requirement item is a key-value pair, such as {"case":{"name":"Like on comment content","steps":[{"action":"click","element":"Dynamic"},{"action":"click","element":"Blank space on screen"},{"action":"click","element":"Like"}]}}. The accuracy and correctness of the requirements are also considered, such as secondary confirmation of the large visual model and coordinate storage, to determine whether the test results meet expectations.

[0020] Visual processing models belong to artificial intelligence algorithm models. For example, the Show UI visual recognition model based on the Qwen2-VL-2B architecture is used for image feature extraction and element recognition. The core processing flow from bottom to top is as follows: 1. Multimodal Input Layer: Receives screenshots of the device interface (visual data) and instruction text (language data), and converts the image into a token sequence through a visual encoder; 2. UI-Guided Feature Interaction Layer: Uses a UI Connectivity Graph to structure the visual tokens, adaptively selecting key visual areas to reduce redundant computation; 3. Interleaved Transformer Layer: Through a visual-language-action interleaved processing mechanism, it fuses visual features with text instructions to dynamically generate operation decisions (such as click location, input content); 4. Action Output Layer: Converts the model decisions into JSON-formatted GUI operation instructions (such as {"action":"click","element":"like"}), realizing end-to-end interface interaction.

[0021] The large language model is a pre-trained language model based on the Transformer architecture, trained on a large amount of text data to learn language knowledge and semantic understanding capabilities. The input layer receives test requirements in natural language form; the encoding layer uses the Transformer's multi-head attention mechanism to encode the input text and extract semantic features; the intermediate layer consists of stacked Transformer blocks to further process and enhance semantic understanding; the output layer generates JSON-formatted test cases and other results based on the encoded semantic information. The Transformer architecture includes components such as multi-head attention, positional encoding, and feedforward neural networks. Multi-head attention can simultaneously focus on different parts of the input text, positional encoding adds positional information to the text, and the feedforward neural network performs non-linear transformations on the encoded information. The number of parameters affects the model's performance and capabilities; for example, QWen has 175 billion parameters. The scale and quality of the training data determine the model's generalization ability and language understanding accuracy; the hidden layer dimension and the number of heads also affect its semantic processing performance.

[0022] The test equipment information includes hardware information such as phone model, operating system, and screen resolution; and software information, covering the device's operating system version and installed application versions. It is divided into three layers: a basic device information layer containing basic identifiers such as device model and brand; a hardware configuration layer, such as processor performance and memory capacity; and a software environment layer, involving the operating system and application versions and configuration information. This is organized in key-value pairs, such as {"Device Model":"iPhone14","Operating System":"iOS16.0","Application Version":"v1.2.3"}, facilitating the system's identification and processing of differentiated information from different devices. Screen resolution determines image display accuracy and affects the visual processing model's recognition of interface elements; operating system version and application version affect application functionality and compatibility, and are key parameters to focus on during testing, as different versions may have different interface layouts and interaction methods.

[0023] S102 processes the mobile UI automation testing requirements based on the visual processing model and the large language model, generates JSON format test cases, and after conversion and integrity checks, generates YAML format test cases for system execution and XMIND format test cases for testers to view and edit.

[0024] In one implementation, the requirements for automated testing of mobile UI are broken down to generate test scenario information, operation step information, and expected result information. In the scenario of using the search function on the APP homepage, the normal network environment covers different network types, such as 4G, 5G, and Wi-Fi. The bandwidth, latency, and other characteristics of different network types will affect the response speed and stability of the search function. For example, there may be signal fluctuations under 4G networks, 5G networks are fast but their coverage may be limited, and Wi-Fi networks are affected by router performance and signal strength. Different search keywords are also crucial. Popular and specific keywords like "mobile phone" can test the system's ability to search for common products; long-tail keywords like "niche retro camera" can test the system's accuracy in searching for specific, niche products; and keywords with multiple meanings, such as "apple" (which can refer to both fruit and Apple brand products), are used to test the system's semantic understanding and accurate matching capabilities.

[0025] Clicking the search box seems simple, but in actual testing, it's crucial to consider the ease of clicking the search box's position. The coordinates of the search box will vary depending on the screen size and resolution of the device. On some small-screen devices, the search box may be difficult to click accurately due to its compact layout; on large-screen devices, the click response area may also be off. When entering keywords, it's necessary to consider not only normal text input but also special character input (such as "@#$%^&*"), mixed Chinese and English input, and extremely long keyword input, to test the system's compatibility and processing capabilities for various input types. When clicking the search button, pay attention to the button's interactive effects, such as whether there is obvious feedback after clicking (e.g., color change, animation effects), and the button's response speed, checking for any issues like no response or excessive delay. Accurately displaying a list of products related to the keywords requires the system to accurately understand the meaning of the keywords and filter out highly matching products from massive amounts of product data. The list sorting should conform to relevance rules, meaning that search results should be sorted comprehensively based on factors such as the degree of matching with the keywords, product sales volume, and user reviews. For example, when searching for "mobile phone," popular brands with high sales volume and good reviews should appear at the top of the search results. Search result loading time should not exceed 3 seconds; this is a crucial metric for search function performance. Excessive loading time severely impacts user experience and leads to user churn. During testing, professional tools should be used to accurately measure loading time, taking into account the impact of different network environments, device performance, and the amount of product data. If loading time exceeds 3 seconds in certain complex network environments or on low-configuration devices, further system performance optimization is necessary.

[0026] The visual processing model and the large language model are adapted and called to generate information on their collaborative processing capabilities. The Show UI, based on Qwen2-VL-2B, is chosen as the visual processing model. Show UI dynamically generates operation decisions (such as click location and input content) by fusing a series of visual features with text commands. It can effectively extract local and global features from the APP interface image and accurately identify the shape, color, and position information of various interface elements, such as buttons, text boxes, and icons. Meanwhile, large language models like QWen, based on the Transformer architecture, possess powerful natural language understanding and generation capabilities. They can understand test requirements described in natural language, such as "Click the search button on the APP homepage, enter the keyword 'phone' and view the search results," and can parse the operation steps and expected goals, providing semantic support for subsequent test case generation.

[0027] In the operational mechanism of collaborative interaction systems, the core essence of dual-model collaborative processing lies in constructing a dynamic and intelligent decision-making mechanism. This mechanism, based on in-depth analysis and logical judgment of key information transmitted by users, uses Boolean values ​​as decision triggers to precisely control model interaction strategies. When the system detects a Boolean value of true, it means that the current interaction scenario needs to update data in real time. At this time, the visual model will fully leverage its perception and computational advantages to accurately locate interface elements and effectively record the newly generated coordinate data, thereby achieving timely response and accurate capture of dynamically changing scenarios. When the Boolean value is false, the system will activate an intelligent reuse strategy, directly calling previously recorded coordinate data to avoid resource consumption caused by repeated calculations, while ensuring the efficiency and smoothness of the interaction process in stable scenarios. This Boolean-based intelligent decision-making logic not only effectively improves the flexibility and adaptability of dual-model collaborative work but also achieves a delicate balance between resource optimization and task execution efficiency, providing solid and reliable technical support for intelligent decision-making in complex interaction scenarios.

[0028] Based on the model's collaborative processing capabilities, the test scenario information, operation step information, and expected result information are processed to generate JSON format test cases. The large language model (QWen) first receives the decomposed test scenario, operation step, and expected result information. For the test scenario "APP homepage search function," it understands that this is a test for a specific function on a specific page of the APP. Regarding the operation steps, the large language model analyzes natural language descriptions such as "click the search box," "enter keywords," and "click the search button," combining its own language understanding capabilities and knowledge reserves to transform them into computer-recognizable operation instructions. In this process, it references the visual model's recognition information of APP interface elements to ensure accurate element positioning in the instructions. For example, if the visual model identifies a specific location of the search box on the APP homepage, the large language model will associate this location information with the "click the search box" operation, making the operation instructions more targeted and accurate.

[0029] The visual model (Show UI) provides crucial UI element recognition support to the large language model throughout the process. It analyzes the app interface, identifying features such as the position, shape, and color of elements like the search box and search button. These recognition results are transmitted to the large language model in data form, helping it to more accurately understand the elements in the test requirements. For example, the visual model determines the boundary coordinates of the search box. When generating JSON-formatted test cases, the large language model can embed this coordinate information appropriately into the "click search box" operation instruction, enhancing the accuracy and executability of the operation and ensuring that the test cases execute correctly in the actual app interface.

[0030] Based on the received information, the large language model begins constructing test cases in JSON format. For the "Test Scenario" section, "APP Homepage Search Function" is directly written as a key-value pair into the JSON structure, i.e., "Test Scenario":"APP Homepage Search Function". In the "Operation Step" section, the large language model transforms each operation step into a sub-object containing an "action" (operation behavior) and either an "element" (operation element) or a "value" (operation value). For example, "Click the search box" is transformed into {"action":"Click","element":"Search Box"}, "Enter Keywords" into {"action":"Enter","value":"Phone"}, and "Click the Search Button" into {"action":"Click","element":"Search Button"}. These sub-objects are then grouped into a list, which serves as the value for the "Operation Step". For the "Expected Results" section, the large language model converts expected content such as "Display Results List" and "Result Sorting" into corresponding key-value pairs based on the expected results information. Finally, it combines "Test Scenario," "Operation Steps," and "Expected Results" into a complete JSON format test case: {"Test Scenario":"APP Homepage Search Function","Operation Steps":[{"action":"Click","element":"Search Box"},{"action":"Input","Value":"Funny Content"},{"action":"Click","element":"Search Button"}],"Expected Results":{"Display Results List":"Related to Funny Content","List Sorting":"Sorted by Relevance","Loading Time":"<=3 seconds"}}.

[0031] This JSON-formatted test case generation through model collaboration offers numerous advantages. It presents test requirements in a standardized, structured format, facilitating reading, parsing, and execution by the computer system. Testers can clearly understand the test content and expected results, simplifying test planning and result verification. Simultaneously, this approach improves the efficiency and accuracy of test case generation, reducing the workload and error rate of manual test case writing. Furthermore, due to the collaborative work of the large language model and the visual model, test cases can better adapt to changes in the app interface. When minor adjustments are made to the app interface, as long as the visual model can recognize the changed elements, the large language model can adjust the test cases accordingly, ensuring the effectiveness and reliability of the tests.

[0032] JSON format test cases undergo structural transformation to generate YAML and XMIND format framework information. This converts JSON test cases into a format easily executed by the system and viewed / edited by testers. When converting JSON data to YAML framework information, emphasis is placed on the system execution flow and parameter settings. When converting to XMIND framework information, a graphical structure is used to facilitate testers' understanding of the test process and scenarios. For example, in an XMIND mind map, "APP Homepage Search Function" serves as the central theme, with "Operation Steps" and "Expected Results" as branch themes, and each specific operation and expected result displayed as a sub-branch.

[0033] Completeness checks are performed on YAML and XMIND format framework information to generate complete YAML test cases for system execution and XMIND test cases for testers to view and edit. In AI vision automated testing systems, performing completeness checks on the converted YAML and XMIND format framework information is a crucial step in ensuring the quality and usability of test cases. The following details the checking process for YAML and XMIND format framework information: Checking and perfecting YAML format framework information: YAML format is used for system execution, and its key fields, such as test scenarios, operation steps, and expected results, are the core basis for test execution. Taking an e-commerce app search function test case as an example, check the "Test Scenario" field to confirm whether it clearly describes accurate content such as "APP homepage search function." If missing, the target scope of the test cannot be determined. For the "Operation Steps" field, check whether it includes specific operation steps such as clicking the search box, entering keywords, and clicking the search button, and whether each operation step has necessary sub-fields such as "action" (operation behavior) and "element" (operation element). If any operation step or sub-field is missing, the system will not be able to fully simulate user operations during execution. In the expected results section, key expected result fields such as "display product list", "list sorting", and "loading time" must be included; otherwise, it is impossible to determine whether the test was successful.

[0034] Besides field completeness, the correctness of data format is also crucial. For element names, check if they are standardized and consistent with the element names recognized by the visual model. For example, if the visual model recognizes the search box as "search_box," but the YAML format shows it as "searchBox," this could lead to inaccurate element location during test case execution, thus affecting test results. After completing field supplementation and format correction, generate the final YAML format test cases for system execution, ensuring the system can accurately operate and verify results according to the test cases.

[0035] The XMIND format framework information should be checked and optimized. Specifically, the XMIND format is used by testers to visually view and edit test cases. A clear and complete mind map structure helps testers understand the test process and scenarios. During the check, first check if the overall structure includes main branches such as test scenarios, operation steps, and expected results. Taking the mind map of an e-commerce app search function test case as an example, the "Test Scenario" branch should clearly describe the relevant scenario information for the app's homepage search function, such as different network environments and searching for different types of keywords. The "Operation Steps" branch should list in detail the specific operation steps such as clicking the search box, entering keywords, and clicking the search button, and the hierarchical relationship of each step should be reasonable, without any chaotic operation order. The "Expected Results" branch should correspondingly display the requirements for the product list, the list sorting rules, and the expected loading time.

[0036] Check that each branch corresponds to the correct test content. For example, in the "Operation Steps" branch, the description of each operation step should be accurate; "Click the search box" should not be written as "Click the search icon." In the "Expected Results" branch, the expected display of the product list should clearly state the specific requirements related to the search keywords, such as "Display products related to mobile phones, including different brands and models." If the mind map is found to have disorganized branches or missing content, adjustments and additions are needed. If the "Enter Keywords" step is missing in the "Operation Steps" branch, it should be added promptly; if the description of the list sorting in the "Expected Results" branch is unclear, it should be clarified as "Sorted in descending order of relevance." After these adjustments and additions, generate XMIND format test cases that testers can view and edit, facilitating test planning, test execution, and further optimization of the test cases.

[0037] S103 processes the test device information based on Appium service and WebDriver to obtain screenshots of the device interface.

[0038] In one implementation, the test device information undergoes device type identification processing to generate device system type information, device resolution information, and device connection status information. Before testing, the system first performs a comprehensive identification and analysis of the test device information. The device type is determined by reading the device's system attributes, configuration files, or using specific device identification tools. For example, for a mobile phone, the system can determine it is an iPhone 14 (belonging to the iOS device type) by querying the device's system settings interface or reading relevant system files. Simultaneously, the device's resolution information is obtained, such as the iPhone 14's resolution of 2532×1170 pixels. The device connection status information is determined by checking the connection method between the device and the test environment (e.g., USB connection, Wi-Fi connection) and whether the connection is stable. If the device is connected via USB and the connection is stable, the device connection status information can be recorded as "USB connection normal". This information is crucial for subsequent testing operations because different types of devices, different resolutions, and connection statuses will affect the test execution method and results.

[0039] The Appium service and WebDriver are initialized and configured to generate service driver readiness information. Appium is an open-source mobile automation testing framework, while WebDriver is an interface for controlling browser operations. Both need to be initialized and configured before using them for testing. When initializing the Appium service, you need to specify its operating port (e.g., the default port 4723) and configure its adaptation parameters for different device systems (iOS or Android). For example, for iOS devices, you need to set the correct BundleID (the application's unique identifier) ​​and Xcode configuration file path. WebDriver initialization requires appropriate settings based on the browser type (e.g., Chrome, Safari), including specifying the browser driver path. After configuration, the Appium service and WebDriver will perform self-checks to confirm that all configurations are correct. If all configurations meet the requirements, service driver readiness information will be generated, indicating that they are ready to receive test commands and interact with the test device.

[0040] Based on service-driven readiness information, the system type, resolution, and connectivity status of the device are processed to obtain screenshots of the device interface. Appium and WebDriver are key tools for capturing device screenshots. During the preparation phase, Appium requires a series of configurations, such as specifying its listening port (usually port 4723 by default) and loading drivers compatible with the iOS device. These drivers are responsible for communicating with the iOS device and understanding and executing various operation commands. Simultaneously, WebDriver also needs initialization settings for the iOS system to ensure it can work collaboratively with Appium to control the device interface interaction. Once Appium and WebDriver have completed their initialization and configuration, and successfully established communication links with each other and the test device, they are in a ready state, waiting to receive instructions to capture screenshots.

[0041] Once the test device is confirmed to be an iOS device, the Appium service will perform operations based on the previously configured iOS-related parameters. These parameters include, but are not limited to, the device's UDID (Unique Device Identifier), iOS system version number, and application's Bundle ID. Using these parameters, the Appium service can accurately identify and connect to the target iOS device. After a successful connection, the Appium service sends a screenshot command to the device. This command is like issuing a "take a picture" task to the device, requesting image data of the currently displayed interface. During the command transmission process, the Appium service utilizes the previously established connection channel to accurately transmit the command to the device.

[0042] WebDriver plays a crucial supporting role in this process. It assists the Appium service in operating on the device's interface to ensure accurate capture of the currently displayed screen. Since iOS devices may have multiple elements and complex interactions, WebDriver can perform preprocessing operations, such as ensuring the interface is stable and free of ongoing animations or loading processes, to prevent incomplete or inaccurate screenshots due to interface instability. Furthermore, WebDriver can make necessary adjustments to the device's interface based on testing requirements, such as scrolling the screen to a specific location to capture the complete interface containing the target element.

[0043] Before capturing a screenshot, the system has already obtained the device's resolution information, such as the iPhone 14's resolution of 2532×1170 pixels. Based on this resolution information, the system will process the screenshot accordingly. On one hand, the system will adjust the screenshot size according to the resolution to ensure that the screenshot can fully display the content of the device interface without cropping or stretching. On the other hand, the system will optimize the screenshot quality based on the resolution. For high-resolution devices, to ensure the clarity and usability of the screenshot in subsequent analysis, the sampling rate of the screenshot may be increased or the image compression ratio adjusted. The screenshot processed in this way can ensure that it contains sufficient detail information without affecting the efficiency of subsequent processing due to excessive file size.

[0044] Once the screenshot operation is successfully completed, the acquired device interface screenshot information will be saved in a specific format. PNG format offers good image quality and compression ratio, making it suitable for storing and transmitting device interface screenshots. The system will store the screenshot in a designated directory, which is usually pre-set in the test environment to facilitate subsequent analysis and processing by the visual processing model. During storage, the system may name the screenshot file. The naming rules can include device information, timestamps, etc., to distinguish and manage screenshots from different devices and at different times. For example, naming it "iOS_device_1_20241001120000.png" indicates that this is a screenshot taken by iOS device number 1 at 12:00 PM on October 1, 2024. Subsequently, the visual processing model will read the screenshot file from this designated directory to perform operations such as element recognition and image analysis, providing important data support for automated testing.

[0045] S104: Based on the visual processing model, the device interface screenshot and element description information are processed to generate the relative coordinates of the elements. After coordinate transformation and operation processing, the operation result is generated.

[0046] In one implementation, a visual processing model is used to analyze screenshots of device interfaces, generating image element distribution information, image color information, and image size information. The visual processing model plays a crucial role in this process. Taking a screenshot of a product display page from an e-commerce app as an example, the model scans the entire image. When generating image element distribution information, it can identify the position and layout of different elements, such as determining the specific location of product images, product names, and price tags within the screenshot. For example, it might determine that the product image is located in the upper left corner of the screenshot, and the product name is below the image. Regarding image color information, the model can analyze the color characteristics of each element, such as the main color tone of the product image and the color of the price tag, which helps to more accurately identify and distinguish elements later. Image size information involves obtaining the width and height of the screenshot, such as a screenshot size of 720 pixels wide and 1280 pixels high. This basic information provides an important basis for subsequent element positioning and analysis.

[0047] The model performs semantic understanding processing on the element description information to generate semantic feature information. Upon receiving the element description information "search button," the visual processing model uses lexical analysis tools from natural language processing to process the text. Lexical analysis breaks down "search button" into independent keywords: "search" and "button." These two keywords are the basic units for understanding the element's semantics, each carrying important semantic clues. "Search" represents the element's core function, implying a connection to finding and retrieving information; "button" clarifies the element's type, indicating it's a control that can be used for interactive operations. The model possesses a pre-built semantic library containing a wealth of semantic knowledge about various interface elements. This knowledge likely comes from learning and analyzing large amounts of interface element data, covering common features, functions, and relationships between different types of elements. Once the keywords "search" and "button" are generated, the model performs a matching search within the semantic library. For "search," the semantic database may record various information related to the search function, such as the operation method and its appearance in different application scenarios. For "button," the semantic database contains common button attributes, such as clickability, common shapes and sizes, and layout patterns in the interface. The model comprehensively analyzes this matched information to further understand the semantics of the "search button." It considers how the search function is combined with the button element, as well as the commonalities and differences in different types of apps. For example, in e-commerce apps, the search button may focus more on the accuracy and speed of product searches; while in news apps, the search button focuses more on content retrieval and display.

[0048] By matching and analyzing with a semantic database, the visual processing model generates semantic feature information for the "search button." This information not only includes the basic semantic concept of the "search button" but also integrates relevant knowledge obtained from the semantic database. First, it clarifies that it is an interactive element used to trigger the search function, which is its core functional feature, indicating that clicking the button will initiate the search process, guiding the system to find relevant content based on the user's input keywords. Second, it determines that it usually has a clickable attribute, a typical interactive characteristic of button elements, meaning that users can interact with the element through touch, click, or other operations. Furthermore, it may be inferred that it carries related icons or text labels. Common search button icons include magnifying glass icons, and text labels may directly display the word "search" or use concise symbols or abbreviations to imply the search function. This semantic feature information provides crucial basis for subsequent matching with image information and determining the element's location. In actual image analysis, the model searches for matching regions in the device interface screenshot based on these features. For example, it searches for elements with clickable attributes (such as an interactive blank area around them) and a magnifying glass icon or the word "search". Combining information such as the color, shape, and position of the image, it can more accurately locate the position of the "search button" in the screenshot, thus providing precise coordinate information for subsequent testing operations.

[0049] Combining the previously generated image-related information and element semantic feature information, the visual processing model begins to determine the relative coordinates of the elements. Continuing with the e-commerce app example, based on the semantic features of the "search button," the model searches for elements matching these features within the image element distribution information. If a button-shaped element located in the upper right corner of the page, with a magnifying glass icon and a color clearly distinguishable from surrounding elements, is found, the model calculates the element's relative coordinates within the entire screenshot, based on the image size information. Relative coordinates are generally based on a corner of the image (such as the upper left corner) as the origin, and their values ​​range from 0 to 1. For example, calculating the relative coordinates of the "search button" as (x = 0.9, y = 0.1) indicates that it occupies 90% of the image width horizontally from the left edge and 10% of the image height vertically from the top edge.

[0050] The system performs coordinate mapping on the relative coordinates of elements to generate the actual pixel coordinates of the device screen. Since different devices have different screen resolutions, it's necessary to convert the relative coordinates to the actual pixel coordinates of the device screen. Assuming the screen resolution of the current test device is 1920×1080 pixels, and the relative coordinates of the "search button" are known to be (x=0.9, y=0.1), the coordinate mapping process multiplies the relative coordinates by the actual width and height of the screen to obtain the actual pixel coordinates. That is, the actual pixel x-coordinate is 0.9×1920=1728 pixels, and the actual pixel y-coordinate is 0.1×1080=108 pixels. This determines the precise position of the "search button" on the device screen, providing accurate coordinates for subsequent simulation operations. After obtaining the actual pixel coordinates of the elements, the system can simulate user operations. For the "search button," the system simulates a click operation. Using an automated testing tool, a simulated click command is sent to the device, with the click position being the previously calculated (1728, 108) pixel point. After the click operation is executed, the system waits for the APP's response and obtains the interface feedback after the operation. If the app responds normally, a search box may pop up, or the user may be redirected to the search results page. The system records these interface changes as operation results. For example, it records that the search box successfully pops up and the cursor inside the search box is blinking, indicating that it is ready to receive input. These operation results are used for subsequent testing and verification to determine whether the app's functionality is working properly.

[0051] S105: Perform screenshot verification on the operation results and generate a test result record.

[0052] In one implementation, after simulating an operation on the app, the system immediately takes a screenshot of the interface. Taking an e-commerce app search function test as an example, after simulating clicking the search button and entering the keyword "mobile phone," the system uses Appium services and related screenshot technology to capture the interface image displayed by the app at that moment. During the screenshot acquisition process, the system ensures that a complete and clear interface image is obtained based on the device's resolution and screen size. The generated screenshot information includes all visible elements on the interface, such as the search results display area, product images, product names, and prices. This information is stored in the form of image data, providing raw material for subsequent comparative analysis.

[0053] The expected result information in the test cases is parsed and processed to generate key feature information of the expected results. The test cases pre-define expected result information, which the system needs to parse to extract key features. For the e-commerce app search function test cases mentioned above, expected results might include "displaying a list of mobile phone-related products," "sorting the list in descending order of relevance," and "loading time not exceeding 3 seconds." The system will analyze these expected results and extract key features. For example, for "displaying a list of mobile phone-related products," key features might include that product images should contain mobile phone-related elements and that the word "mobile phone" should appear in the product name; for "sorting the list in descending order of relevance," key features involve the logical judgment basis for product sorting; and for "loading time not exceeding 3 seconds," the key feature is the time threshold. This key feature information will serve as an important standard for judging whether the interface after operation meets expectations.

[0054] The visual processing model plays a crucial role in the comparison process. It first performs image analysis on the screenshot of the interface after the operation, identifying various elements such as product images, product names, and list layouts. Combining relevant image analysis algorithms, it precisely analyzes the content, position, and quantity of these elements. Taking the screenshot of the interface after searching for "mobile phone" on an e-commerce app as an example, the visual processing model uses image recognition technology to extract feature information from the product image and compares it with a pre-defined database of mobile phone-related features. This database contains typical image features of different brands and models of mobile phones; similarity is compared to determine whether the product image is indeed related to mobile phones. For product names, the visual processing model uses Optical Character Recognition (OCR) technology to extract the text from the image and then performs text analysis to check for the presence of the word "mobile phone." When checking element positions, the model determines the specific position of each product element in the interface based on the image's coordinate system and compares it with the expected layout to determine if it conforms to design specifications. For product images, the visual processing model makes judgments from multiple dimensions. It not only analyzes the overall shape, color, and other appearance features of the image but also delves into the details of the image. For example, the phone's screen, camera, and brand logo are all important criteria for judgment. If the image shows other items, such as a computer or books, with extremely low similarity to features in the phone's feature database, then the product image is considered unrelated to the phone.

[0055] After extracting the text from the product name, the visual processing model performs semantic analysis. Besides simply determining if the keyword "mobile phone" is included, it considers synonyms and related terms such as "mobile phone" and "handheld terminal." Simultaneously, the model filters out distracting information, such as promotional slogans and model suffixes, to ensure accurate identification of whether a product is related to mobile phones. For example, a product name like "fashionable backpack" clearly doesn't contain any mobile phone-related words, which is unexpected. For list sorting, the system uses a relevance algorithm to determine if it should be sorted in descending order. This algorithm considers multiple factors, such as keyword matching, product sales, and user reviews. After searching for "mobile phone," highly relevant products should appear first. The system extracts relevant data for each product, simulates sorting according to the algorithm, and then compares this simulation with the actual displayed list. If, in the actual list, products with low sales and poor keyword matching for "mobile phone" appear before products with high sales and good matching, then the list sorting does not conform to the descending order rule.

[0056] To determine if the loading time meets the requirements, the system precisely records the operation time and screenshot time. The system starts timing when the user triggers a search and stops timing when a screenshot of the interface after the operation is captured, thus obtaining the loading time. If the set loading time threshold is 3 seconds, and the actual loading time exceeds 3 seconds, it indicates that the loading time does not meet expectations. This may be due to network latency, insufficient app code optimization, or other reasons, requiring further investigation. During the above checks, if any key features of the interface after the operation are found that do not conform to the expected results, these parts will be identified as discrepancy features. For example, if a product image shows a tablet computer but appears in the search results for "phone," then this product image unrelated to phones is a discrepancy feature; or if the products in the list are sorted haphazardly, not in descending order of relevance, this sorting error is also a discrepancy feature; a loading time exceeding 3 seconds is also a discrepancy feature. These discrepancy features are important criteria for judging test results, and the system records them in detail for subsequent analysis of problems, helping developers to make targeted optimizations and improvements.

[0057] Based on the generated difference feature information, the system makes a judgment to determine the test result. If the difference feature information is empty, meaning the interface after operation fully matches the key features of the expected result, the system determines that the test has passed; otherwise, if difference feature information exists, the test is determined to have failed. For example, if the comparative analysis finds that some products in the search results list are unrelated to mobile phones, the system will generate a test failure judgment message, clearly pointing out the problem that "there are products unrelated to mobile phones in the search results." The judgment process can also be graded according to the severity of the difference, such as severe difference (e.g., core functions not implemented) and general difference (e.g., partial abnormality in interface display), so that testers can better understand the severity of the problem.

[0058] The system records and organizes test results based on the judgment information, generating test result records. The records include the test case name, execution time, operation steps, expected result, actual result, whether the test passed or failed, and specific differences (if the test failed). For the above e-commerce app search function test, the test result record is as follows:

[0059]

[0060] These test results records provide testers with clear test feedback, helping them to quickly locate problems and carry out subsequent problem fixing and optimization work.

[0061] S106 processes the test result records and YAML and XMIND format test cases to generate test report information.

[0062] In one implementation, the test result records are parsed to generate test execution status information, test pass / fail count information, and exception information. The test result records contain detailed information about each test execution. Taking multiple test records of an e-commerce app's search function as an example, the system reads these records and parses the operation steps, actual operation results, and whether the test passed or failed. If, in a test, the search operation was executed according to the test case, and the actual result meets expectations (e.g., displaying the correct product list, correct sorting, and loading time within the specified range), the system marks the execution status of this test as "successful"; if the actual result does not meet expectations (e.g., empty search results, disordered list sorting, or loading timeout), it is marked as "failed." By statistically analyzing all test result records, the system can determine the test pass / fail count; for example, in 100 search function tests, 90 passed and 10 failed. Simultaneously, for exceptions occurring during the test, such as app crashes or interface lag preventing operation completion, the system extracts relevant information and records it in detail, for example, "During the 50th test, the app crashed."

[0063] YAML format test cases define the specific content and expected results of the test. Taking the e-commerce app search function test case as an example, the detailed test steps can be extracted from the YAML file, such as "Click the search box -> Enter the keyword 'mobile phone' -> Click the search button". Expected operation results include "Display a list of mobile phone-related products, sorted in descending order of relevance, with a loading time of no more than 3 seconds". Test environment configuration information includes the device information used during testing (e.g., device model is iPhone 14, operating system is iOS 16.0), network environment (e.g., Wi-Fi connection, network bandwidth is 100Mbps), etc. This information is crucial for understanding the execution conditions and expected results of the test cases, providing a benchmark for subsequent test result analysis. Visual information extraction processing is performed on XMIND format test cases to generate test case logical structure information and scenario description information. XMIND format test cases graphically display the test process and scenarios. In the test cases for the search function of an e-commerce app, the logical structure information of the test cases illustrates the relationships between the various test steps. For example, "clicking the search box" is a prerequisite for "entering keywords," and "entering keywords" is a prerequisite for "clicking the search button." These steps together constitute the logical flow of the search function test. The scenario description information details the specific test scenario, such as "performing a search operation on the app's homepage to test the search function under different network environments (4G, 5G, Wi-Fi)," enabling testers to intuitively understand the background and purpose of the test.

[0064] The extracted information is comprehensively analyzed to generate a comprehensive test report. For example, in testing the search function of an e-commerce app, the test execution status information shows that most tests were successful, but some failed; the number of failed tests indicates a failure rate of 10%; the anomaly information indicates that some failures were due to network instability causing excessively long loading times; the test step details and expected operation results clarify the specific requirements and expectations of the test; the test environment configuration information shows that the test results differ under different network environments; and the test case logic structure information and scenario description information help understand the overall test process and background. Based on this information, the test report concludes that: "The e-commerce app's search function has good overall stability, but in some poor network environments, there is a problem of excessively long loading times, affecting user experience. It is recommended to further optimize the network request mechanism to improve the response speed of the search function under different network environments." The test report may also display the proportion of failed tests and the comparison of test results under different network environments in the form of charts, making the test results more intuitive and easier to understand.

[0065] This application focuses on automated testing methods and systems based on AI vision, with a core focus on improving the efficiency and accuracy of automated testing for mobile UI using various technologies. It first acquires test requirements, models, and device information. Then, it processes the requirements using visual and large language models to generate test cases in various formats. Device information is processed through Appium services and WebDriver to obtain screenshots of the interface. The visual processing model analyzes the screenshots and element descriptions, deriving relative coordinates and converting them to actual pixel coordinates to simulate operations and generate results. The results are then verified against screenshots, and compared with expected results to generate test result records. Finally, a comprehensive test report is generated by integrating the test result records, key information from YAML and XMIND format test cases. This approach solves problems such as easy failure in element location and difficulty in multi-platform adaptation in traditional testing methods, enabling natural language description of test cases and cross-platform testing, thus improving the stability, efficiency, and coverage of testing.

[0066] In one implementation, such as Figure 3 As shown, this application also provides an automated testing device for AI vision, comprising:

[0067] The acquisition module 301 is used to acquire mobile UI automation testing requirements, visual processing models, large language models, and test device information;

[0068] Processing module 302 is used to process mobile UI automation testing requirements based on visual processing models and large language models, generate JSON format test cases, and after conversion and integrity checks, generate YAML format test cases for system execution and XMIND format test cases for testers to view and edit; process test device information based on Appium service and WebDriver to obtain device interface screenshot information; process device interface screenshots and element description information based on visual processing models to generate element relative coordinates, and generate operation results after coordinate transformation and operation processing; perform screenshot verification processing on operation results to generate test result records; process test result records and YAML format test cases and XMIND format test cases to generate test report information.

[0069] The various embodiments in this application are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments of automated testing methods, electronic devices, electronic devices, and readable storage media for evaluating AI vision are basically similar to the above-described embodiments of automated testing methods for AI vision, and therefore the descriptions are relatively simple. Relevant parts can be referred to in the descriptions of the above-described embodiments of automated testing methods for AI vision.

Claims

1. An automated testing method for AI vision, characterized in that, include: Obtain mobile UI automation testing requirements, visual processing models, large language models, and testing device information; The system processes mobile UI automation testing requirements based on visual processing models and large language models, generates JSON format test cases, and after conversion and integrity checks, generates YAML format test cases for system execution and XMIND format test cases for testers to view and edit. The test device information is processed using Appium service and WebDriver to obtain screenshots of the device interface; Based on the visual processing model, the device interface screenshots and element description information are processed to generate relative coordinates of the elements. After coordinate transformation and operation processing, the operation results are generated. The operation results are verified by taking screenshots, and test result records are generated. The test results records and YAML and XMIND format test cases are processed to generate test report information.

2. The method as described in claim 1, characterized in that, This paper processes mobile UI automation testing requirements based on visual processing models and large language models, generating JSON format test cases. After conversion and integrity checks, it generates YAML format test cases for system execution and XMIND format test cases for testers to view and edit. The requirements for automated testing of mobile UI are broken down and processed to generate test scenario information, operation step information and expected result information; Perform call adaptation processing on the visual processing model and the large language model to generate information on the collaborative processing capabilities of the models; Based on the model's collaborative processing capabilities, test scenario information, operation step information, and expected result information are processed to generate JSON format test cases; Perform structural conversion on JSON format test cases to generate YAML and XMIND format framework information; Perform integrity checks on YAML and XMIND format framework information to generate complete YAML format test cases for system execution and XMIND format test cases for testers to view and edit.

3. The method as described in claim 1, characterized in that, The test device information is processed using Appium services and WebDriver to obtain device interface screenshots, including: The test equipment information is processed to identify the equipment type, generating equipment system type information, equipment resolution information, and equipment connection status information. Initialize and configure the Appium service and WebDriver, and generate service driver readiness information; Based on service-driven readiness information, the device system type information, device resolution information, and device connection status information are processed to obtain device interface screenshot information.

4. The method as described in claim 1, characterized in that, Based on a visual processing model, screenshots of the device interface and element descriptions are processed to generate relative coordinates of the elements. After coordinate transformation and manipulation, operation results are generated, including: Based on the visual processing model, the device interface screenshot is processed by image parsing to generate image element distribution information, image color information and image size information; Perform semantic understanding processing on element description information to generate element semantic feature information; Based on the semantic features of the elements, the image element distribution information, image color information, and image size information are processed to generate relative coordinates of the elements; Perform coordinate mapping on the relative coordinates of the elements to generate the actual pixel coordinates of the device screen; The simulation operation is performed based on the actual pixel coordinates of the device screen to generate the operation result.

5. The method as described in claim 4, characterized in that, The operation results are verified by taking screenshots, and test result records are generated, including: The operation results are captured and processed to generate screenshot information of the interface after the operation. The expected result information in the test cases is parsed and processed to generate key feature information of the expected results. Based on the key feature information of the expected results, the screenshot information of the interface after the operation is compared and analyzed to generate difference feature information. The differential feature information is processed to generate test result determination information; Based on the test result judgment information, record and organize the information to generate test result records.

6. The method as described in claim 1, characterized in that, The test results records and YAML and XMIND format test cases are processed to generate test report information, including: The test result records are parsed and processed to generate test execution status information, the number of test passes and failures, and abnormal situation information. Extract and process key information from YAML format test cases to generate test step details, expected operation results, and test environment configuration information; Visual information extraction and processing are performed on XMIND format test cases to generate test case logical structure information and scenario description information; Test report information is generated by comprehensively analyzing and processing test execution status information, number of test successes and failures, exception information, test step details, expected operation results, test environment configuration information, test case logical structure information, and scenario description information.

7. An automated testing device for AI vision, characterized in that, The device includes: The acquisition module is used to acquire mobile UI automation testing requirements, visual processing models, large language models, and testing device information. The processing module is used to process mobile UI automation testing requirements based on visual processing models and large language models, generate JSON format test cases, and after conversion and integrity checks, generate YAML format test cases for system execution and XMIND format test cases for testers to view and edit. It processes test device information based on Appium services and WebDriver to obtain device interface screenshots; processes device interface screenshots and element description information based on the visual processing model to generate element relative coordinates, and after coordinate transformation and operation processing, generates operation results; performs screenshot verification processing on the operation results to generate test result records; and processes the test result records, YAML format test cases, and XMIND format test cases to generate test report information.

8. An electronic device, characterized in that, include: First processor; and memory for storing executable instructions of the first processor; The first processor is configured to execute the automated testing method for AI vision according to any one of claims 1 to 6 by executing the executable instructions.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the second processor, it implements the automated testing method for AI vision as described in any one of claims 1 to 6.

Images