An AI vision recognition-based mobile terminal full-scene automatic test system and a test method thereof

Abstract

The application relates to the technical field of mobile terminal automatic testing, in particular to a mobile terminal full-scene automatic testing system based on AI visual identification and a testing method thereof, which comprises a backend control layer, an automatic driving layer, an AI visual identification layer, a time execution layer, a cross-platform adaptation layer and a tool support layer. The application reduces the dependence on UI element IDs in the mode of AI visual identification, does not need to modify scripts in iteration, can directly realize version control, improves the maintenance and testing efficiency, and has high positioning accuracy and compatibility.

Description

Technical Field

[0001] This invention relates to the field of mobile terminal automatic testing technology, specifically to a mobile terminal full-scenario automatic testing system and testing method based on AI visual recognition. Background Technology

[0002] Mobile automated testing uses scripts / tools to automatically verify the functionality, compatibility, and stability of apps or mobile websites on mobile devices such as phones and tablets, replacing manual testing. Existing testing technologies have the following shortcomings: 1. Existing automated testing technologies for mobile devices are highly dependent on UI element IDs. When frequent application iterations lead to changes in the UI structure, a large number of test scripts need to be modified, resulting in high maintenance costs.

[0003] 2. Test scripts are managed in a scattered manner, version control is chaotic, team collaboration is inefficient, and it is difficult to achieve unified scheduling of multiple devices.

[0004] 3. The element location method is too simple, and the accuracy of location is poor when facing complex UI scenarios (such as CAPTCHA recognition and nested elements), which can easily lead to test failure.

[0005] 4. Insufficient cross-platform compatibility, making it difficult to simultaneously support different versions of Android / iOS and various devices, resulting in limited testing coverage.

[0006] 5. The test execution process lacks centralized monitoring, making it difficult to trace the cause of failure, and the efficiency of concurrent execution on multiple devices is low.

[0007] 6. Traditional testing solutions do not adequately support the joint testing of apps and mini-programs, and cannot meet the needs of full-scenario and full-link testing.

[0008] Based on the above reasons, this invention designs a mobile terminal full-scene automated testing system and its testing method based on AI visual recognition. By using AI visual recognition, the dependence on UI element IDs is reduced, iteration does not require script modification, version control can be directly realized, maintenance and testing efficiency is improved, and it has high positioning accuracy and compatibility. Summary of the Invention

[0009] The purpose of this invention is to overcome the shortcomings of the prior art and provide a mobile terminal full-scene automated testing system and testing method based on AI visual recognition. By using AI visual recognition, the system reduces the dependence on UI element IDs, iterates without modifying the script, can directly achieve version control, improve maintenance and testing efficiency, and has high positioning accuracy and compatibility.

[0010] To achieve the above objectives, this invention provides a mobile terminal full-scenario automated testing system based on AI visual recognition, comprising the following levels: Build the backend control layer: The core modules are developed based on a backend language. The core modules include a script management module, a distribution module, an execution control module, and a monitoring module, and support concurrent management of multiple devices. Building an automation-driven layer: An automation framework integrating Appium and Selenium, adapted to ADB and XCTest device communication protocols, used to achieve automated control of Android and iOS dual-platform mobile devices; Deploying an AI visual recognition layer: Integrating object detection models and OCR technology, we developed a UI element recognition module, a coordinate calculation module, and an OCR enhancement module to achieve ID-free element localization. Development time execution layer: The design of the event parsing engine supports basic event types such as click, input, swipe, wait, and assertion, and also supports nested event configuration to achieve seamless switching between the APP and the mini program. Building a cross-platform adaptation layer: Collect UI difference data from different systems and devices, and design a unified test interface and dynamic adaptation algorithm to achieve standardized element operations across applications and devices; Configuration tool support layer: It integrates basic functions such as screenshot recording, real-time log uploading, asynchronous waiting, and intelligent retries to support test execution.

[0011] The script management module is used to store and version control scripts; the distribution module is used to dynamically schedule scripts based on device status and testing requirements; the execution control module remotely controls the pause, resume, and termination of scripts; and the monitoring module collects execution status in real time.

[0012] The UI element recognition module includes screenshot analysis and model matching; the coordinate calculation module is used to automatically locate the center point of elements; and the OCR enhancement module supports CAPTCHA and text extraction.

[0013] The screenshot records include key nodes and failure context.

[0014] Backend languages ​​include Python, Java, and Go.

[0015] The automation frameworks for Appium and Selenium can be used for Espresso and XCUITest.

[0016] A testing method for a mobile terminal full-scenario automated testing system based on AI visual recognition includes the following steps: S1, Script Configuration: Define the test process through the configuration file, specify the event type, execution order, element identification rules and assertion conditions, and upload it to the backend control layer for unified storage; S2, Device Access: Mobile devices establish a connection with the system via ADB or XCTest, and the backend control layer automatically obtains information such as device model, system version, and idle status. S3, Script Distribution: The backend dynamically distributes the matched JSON test scripts to the target devices based on test requirements and device status; S4, Element Location: The device takes a screenshot of the current interface. The AI ​​visual recognition layer identifies the target UI elements through the YOLO model and extracts their text information using OCR technology. It supports multiple location methods such as text recognition, name recognition, and nested search, and automatically calculates the coordinates of the element's center point. S5, Event Execution: The event execution layer parses script instructions and executes corresponding operations. Non-blocking waiting is achieved through an asynchronous countdown timer, and the dynamic element waiting mechanism avoids hard-coding of waiting time, thereby improving execution efficiency. S6, Process Monitoring: During the test, the tool support layer automatically saves screenshots of key nodes and uploads execution logs to the backend in real time. The backend control layer visualizes the execution status and supports remote intervention. S7, Result Processing: After the test is completed, the system automatically generates a multi-dimensional report, performs intelligent retries for failed test cases, and stores all execution records, screenshots, and logs in the backend.

[0017] Key node screenshots include context screenshots retained when a test fails.

[0018] The multi-dimensional report includes execution status, time taken, and reasons for failure.

[0019] Configuration files can be in JSON, XML, or YAML format.

[0020] Compared with the prior art, the present invention has the following beneficial effects: 1. Reduced maintenance costs: This invention reduces reliance on UI element IDs through AI visual recognition, eliminating the need for extensive script modifications during application iterations. Centralized backend management enables version control, improving maintenance efficiency.

[0021] 2. Improved testing efficiency: This invention combines concurrent execution on multiple devices with an asynchronous execution mechanism, which significantly shortens the testing cycle; and improves the success rate of test case execution by combining dynamic waiting with intelligent retry strategies.

[0022] 3. Enhanced compatibility: This invention supports all Android / iOS platforms and different devices and system versions. The cross-platform adaptation layer covers mainstream mobile devices, and the testing coverage is increased by 80%.

[0023] 4. Improved positioning accuracy: This invention uses the YOLO model combined with OCR technology to solve the positioning problem in complex scenarios (such as CAPTCHAs and nested elements), and the positioning error is controlled at the pixel level, avoiding human error.

[0024] 5. Optimized management and collaboration: The centralized backend control of this invention enables unified script management, visual monitoring of execution status, and centralized storage of results, facilitating team sharing and collaboration, resource scheduling, and problem tracing.

[0025] 6. Supports full-scenario testing: This invention can cover testing of apps, mini-programs and the combination of the two, adapt to continuous integration (CI / CD), regression testing and full-link testing requirements, support agile development and shorten product iteration cycle.

[0026] 7. Flexible Expansion: This invention adopts a modular architecture design, which can quickly add test event types and adapt to new application types to meet the mobile application testing needs of different industries. Detailed Implementation

[0027] This embodiment provides a mobile terminal full-scene automated testing system and its testing method based on AI visual recognition, including the following: I. System Architecture Setup (Implementation Steps by Level): 1. Build the backend control layer: Develop core modules based on Python, including a script management module (implementing script storage and version control), a distribution module (dynamically scheduling based on device status and test requirements), an execution control module (remotely controlling script pause / resume / termination), and a monitoring module (real-time collection of execution status), supporting concurrent management of multiple devices; 2. Build an automation driver layer: Integrate the Appium and Selenium automation frameworks, adapt to ADB (Android Debug Bridge) and XCTest (iOS) device communication protocols, and realize automated control of Android and iOS dual-platform mobile devices; 3. Deploy the AI ​​visual recognition layer: Integrate the YOLO object detection model and PaddleOCR technology to develop a UI element recognition module (screenshot analysis + model matching), a coordinate calculation module (automatically locate the element center point), and an OCR enhancement module (support CAPTCHA and text extraction) to achieve ID-free element positioning; 4. Develop the event execution layer: Design an event parsing engine that supports basic event types such as click, input, swipe, wait, and assertion, while also supporting nested event configuration to achieve seamless switching between the APP and the mini program; 5. Build a cross-platform adaptation layer: Collect UI difference data from different systems and devices, design a unified test interface and dynamic adaptation algorithm, and realize standardized element operations across applications and devices; 6. Configuration tool support layer: Integrates basic functions such as screenshot recording (key nodes + failure context), real-time log uploading, asynchronous waiting, and intelligent retries to support test execution.

[0028] In addition, backend technology alternatives include Java, Go, and other backend languages ​​besides Python, which can be used to implement backend control layer functions while maintaining consistency in core logic (script management, distribution, and monitoring).

[0029] Visual recognition model replacement: The YOLO model can be replaced with object detection models such as Faster R-CNN and SSD, and OCR technology can be replaced with mature engines such as Tesseract OCR, without changing the core localization logic of "visual recognition combined with text extraction"; Automation framework alternatives: Appium and Selenium can be replaced with platform-specific frameworks such as Espresso (Android) and XCUITest (iOS). By adjusting the interface through the adaptation layer, cross-platform automation control capabilities can be maintained. II. Test Execution Process (by Operation Steps): 1. Script configuration: Define the test process through a JSON format configuration file, specifying event types, execution order, element recognition rules, assertion conditions, etc., and upload it to the backend control layer for unified storage; 2. Device Access: Mobile devices (Android / iOS) establish a connection with the system via ADB or XCTest, and the backend control layer automatically obtains information such as device model, system version, and idle status; 3. Script distribution: The backend dynamically distributes the matching JSON test scripts to the target devices based on test requirements (such as functional testing and compatibility testing) and device status; 4. Element localization: The device takes a screenshot of the current interface, and the AI ​​visual recognition layer identifies the target UI element through the YOLO model. Combined with OCR technology, it extracts text information and supports multiple localization methods such as text recognition, name recognition, and nested search. It automatically calculates the coordinates of the element's center point. 5. Event Execution: The event execution layer parses script instructions and executes corresponding operations. Non-blocking waiting is achieved through an asynchronous countdown timer, and the dynamic element waiting mechanism avoids hard-coding of waiting time, thus improving execution efficiency. 6. Process monitoring: During the test, the tool support layer automatically saves screenshots of key nodes (and retains context screenshots in case of failure), uploads execution logs to the backend in real time, and the backend control layer visualizes the execution status and supports remote intervention; 7. Results Processing: After the test is completed, the system automatically generates a multi-dimensional report (including execution status, time consumption, and failure reasons), performs intelligent retries for failed test cases, and stores all execution records, screenshots, and logs in the backend.

[0030] In addition, script format replacement: JSON configuration files can be replaced with common formats such as XML and YAML. Only the backend script parsing module needs to be modified, which does not affect the centralized management and distribution functions. Functional expansion implementation: Add a performance monitoring module (collecting CPU and memory usage data) or an AI risk prediction module (identifying potential test problems in advance) to the existing architecture. Deployment method alternative: Supports cloud deployment architecture, expands the capacity of the backend control layer through cloud servers, and supports thousands of devices to access the test at the same time.

[0031] The above are merely preferred embodiments of the present invention, intended only to aid in understanding the method and core ideas of this application. The scope of protection of the present invention is not limited to the above embodiments; all technical solutions falling within the scope of the present invention's concept are within its protection. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should also be considered within the scope of protection of the present invention.

[0032] This invention comprehensively addresses the shortcomings of existing mobile automated testing technologies, such as high dependence on UI element IDs, high maintenance costs, low collaboration efficiency, single positioning methods, low concurrent execution efficiency, and insufficient cross-platform adaptability. By using AI visual recognition, it reduces the dependence on UI element IDs, eliminates the need to modify scripts for iteration, enables direct version control, improves maintenance and testing efficiency, and has high positioning accuracy and compatibility.

Claims

1. A mobile terminal full-scenario automated testing system based on AI visual recognition, characterized in that, Includes the following levels: Build the backend control layer: The core modules are developed based on a backend language. These core modules include a script management module, a distribution module, an execution control module, and a monitoring module, and support concurrent management of multiple devices. Building an automation-driven layer: An automation framework integrating Appium and Selenium, adapted to ADB and XCTest device communication protocols, used to achieve automated control of Android and iOS dual-platform mobile devices; Deploying an AI visual recognition layer: Integrating object detection models and OCR technology, we developed a UI element recognition module, a coordinate calculation module, and an OCR enhancement module to achieve ID-free element localization. Development time execution layer: The design of the event parsing engine supports basic event types such as click, input, swipe, wait, and assertion, and also supports nested event configuration to achieve seamless switching between the APP and the mini program. Building a cross-platform adaptation layer: Collect UI difference data from different systems and devices, and design a unified test interface and dynamic adaptation algorithm to achieve standardized element operations across applications and devices; Configuration tool support layer: It integrates basic functions such as screenshot recording, real-time log uploading, asynchronous waiting, and intelligent retries to support test execution.

2. The mobile terminal full-scene automated testing method based on AI visual recognition according to claim 1, characterized in that, The script management module is used to store and version control scripts; the distribution module is used to dynamically schedule scripts based on device status and testing requirements; the execution control module remotely controls the pause, continuation, and termination of scripts; and the monitoring module collects execution status in real time.

3. The automated testing method for mobile terminals across all scenarios based on AI visual recognition according to claim 1, characterized in that, The UI element recognition module includes screenshot analysis and model matching; the coordinate calculation module is used to automatically locate the center point of the element; and the OCR enhancement module supports CAPTCHA and text extraction.

4. The automated testing method for mobile terminals across all scenarios based on AI visual recognition according to claim 1, characterized in that, The screenshot records include key nodes and failure context.

5. The automated testing method for mobile terminals across all scenarios based on AI visual recognition according to claim 1, characterized in that, The backend languages ​​include Python, Java, and Go.

6. The mobile terminal full-scene automated testing method based on AI visual recognition according to claim 1, characterized in that, The automation frameworks for Appium and Selenium can be Espresso and XCUITest.

7. A testing method for a mobile terminal full-scene automated testing system based on AI visual recognition as described in claim 1, characterized in that, Includes the following steps: S1, Script Configuration: Define the test process through the configuration file, specify the event type, execution order, element identification rules and assertion conditions, and upload it to the backend control layer for unified storage; S2, Device Access: The mobile device establishes a connection with the system through the ADB or the XCTest, and the backend control layer automatically obtains information such as device model, system version, and idle status. S3, Script Distribution: The backend dynamically distributes the matched JSON test scripts to the target devices based on test requirements and device status; S4, Element localization: The device takes a screenshot of the current interface. The AI ​​visual recognition layer identifies the target UI elements through the YOLO model and extracts their text information by combining OCR technology. It supports multiple localization methods such as text recognition, name recognition, and nested search, and automatically calculates the coordinates of the element's center point. S5, Event Execution: The event execution layer parses script instructions and executes corresponding operations. Non-blocking waiting is achieved through an asynchronous countdown timer, and the dynamic element waiting mechanism avoids hard-coding of waiting time, thereby improving execution efficiency. S6, Process Monitoring: During the test, the tool's support layer automatically saves screenshots of key nodes, uploads execution logs to the backend in real time, and the backend control layer visualizes the execution status and supports remote intervention. S7, Result Processing: After the test is completed, the system automatically generates a multi-dimensional report, performs intelligent retries for failed test cases, and stores all execution records, screenshots, and logs in the backend.

8. The testing method of the mobile terminal full-scene automated testing system based on AI visual recognition according to claim 7, characterized in that, The key node screenshots include context screenshots retained when the test fails.

9. The testing method of the mobile terminal full-scene automated testing system based on AI visual recognition according to claim 7, characterized in that, The multi-dimensional report includes execution status, time taken, and reasons for failure.

10. The testing method of the mobile terminal full-scene automated testing system based on AI visual recognition according to claim 7, characterized in that, The configuration file can be in JSON, XML, or YAML format.