Cross-platform UI automation system based on graphical interface

Abstract

The invention relates to the technical field of software automation testing, in particular to a cross-platform UI automation system based on a graphical interface, which comprises a core framework unit, a script engine unit, a graphical interface tool unit, an expansion plug-in and library unit and a document and support unit, the core framework unit is used for providing basic functions and interfaces, so that developers can perform UI automatic operation on different platforms; the script engine unit is used for allowing a developer to write an automatic script so as to define a specific automatic operation process; the graphical interface tool unit is used for providing a visual operation interface so as to create, edit and manage visualization of automated test scripts and test cases; the expansion plug-in and library unit is used for providing additional functions and support; the script engine built in the system supports various programming languages and is equipped with a powerful debugging tool, so that developers can easily compile, debug and execute automatic scripts, and the development efficiency is greatly improved.

Description

Technical Field

[0001] The present invention relates to the technical field of software automated testing, and in particular to a cross-platform UI automated system based on a graphical interface. Background Art

[0002] UI automation is an automated testing method that simulates manual operation of the user UI interface and implements automatic operation and verification in code.

[0003] After checking the publication number: CN117931611A, a cross-platform UI automation testing method is disclosed. This technology discloses "a cross-platform UI automation testing method, the purpose is to improve the UI automation testing efficiency under the background of cross-platform testing requirements, and in this process reduce the time of manual intervention, and realize the stable execution of the test in the unmanned monitoring mode. The present invention is implemented by the following technical solutions: 1) Incorporating the test process parameters into the configuration item management layer; 2) The test control layer is operated by preset keywords to obtain the test scope and the system version to be tested, and the interface communication technology is used to send a start command to the business execution layer; 3) The business execution layer starts the Windows desktop program or Android program test case script according to the instructions of the test control layer, and configures the process parameters in the item management layer according to the key business parameters involved in the test scope; 4) The result output layer obtains the test case execution information, and monitors the test execution process through the assertion mechanism, writes the test results into EXCEL or HTML synchronous output, and automatically sends emails" and other technical solutions, with "By integrating the UI automation testing tools of Windows desktop programs and Android programs into a unified scheduling platform, the problem of switching test tools for testing different platforms is solved, and cross-platform UI automation testing is realized" and other technical effects;

[0004] Since script compatibility issues stem from the grammatical and logical differences between programming languages, cross-language use requires cumbersome conversion, which affects execution efficiency. In complex execution environments, scripts may fail to execute due to environmental or configuration issues, reducing efficiency. During debugging, the lack of efficient tools makes it difficult to locate errors, prolongs the development cycle, and increases project risks. In addition, script writing and management are inefficient, and traditional editors lack intelligent functions, making writing cumbersome and error-prone. Summary of the invention

[0005] In view of the deficiencies in the prior art, the present invention provides a cross-platform UI automation system based on a graphical interface. The system has a built-in script engine that supports multiple programming languages ​​and is equipped with powerful debugging tools, so that developers can easily write, debug and execute automation scripts, greatly improving development efficiency; and the graphical interface tool provides an intuitive operating interface, reduces the difficulty of use, and enables developers to complete testing tasks more quickly.

[0006] To achieve the above objectives, the present invention is implemented through the following technical solutions: A cross-platform UI automation system based on a graphical interface, including a core framework unit, a script engine unit, a graphical interface tool unit, an extension plug-in and library unit, and a document and support unit;

[0007] The core framework unit is used to provide basic functions and interfaces, allowing developers to perform UI automation operations on different platforms;

[0008] The script engine unit is used to allow developers to write automation scripts to define specific automation operation processes;

[0009] Graphical interface tool unit, used to provide an intuitive operating interface to create, edit and manage automated test scripts and visualization of test cases;

[0010] Extension plugins and library units for providing additional functionality and support, as well as implementation of custom script commands and tools;

[0011] The documentation and support unit is used to provide system usage guides, API references, troubleshooting methods, and user support services.

[0012] Preferably, the core framework unit comprises:

[0013] The cross-platform adaptation layer provides adaptation for UI element recognition, event processing, input simulation, etc. on different platforms, so that the system can achieve consistent automated operations on different operating systems;

[0014] The UI element recognition module obtains the properties and status information of UI elements through image recognition and control tree traversal, providing basic data for subsequent automated operations;

[0015] The event processing module monitors and processes UI events, triggers corresponding automation operations and executes corresponding script logic according to event types and event parameters;

[0016] Input simulation module simulates user input behaviors, such as keyboard input, mouse clicks, and touch slides.

[0017] Preferably, the script engine unit includes:

[0018] Script parser: parses the automation scripts written by developers and converts them into instructions that can be executed by the system, ensuring the correctness and executability of the scripts and providing basic instructions for automated operations;

[0019] The script execution environment provides an isolated execution environment for executing automated scripts, protecting the system core code from script errors, while providing the resources and interfaces required for script execution.

[0020] Preferably, the script engine unit also includes a script debugging tool and is used to provide debugging functions such as breakpoints, single-step execution, and variable viewing, so as to quickly locate and solve problems when writing and debugging automation scripts.

[0021] Preferably, the graphical interface tool unit includes:

[0022] Script editor, which provides code editing functions and supports syntax highlighting and automatic completion;

[0023] The control selector provides a visual control selection function to quickly locate and select UI elements on the screen.

[0024] Preferably, the graphical interface tool unit also includes a log viewer and is used to display the execution log and error information of the automation script.

[0025] Preferably, the extension plug-in and library unit include:

[0026] The control library provides support and recognition capabilities for common UI controls, expanding the control recognition range of the system;

[0027] Script library, providing commonly used automation scripts and function libraries;

[0028] Custom plugins, allowing development of custom plugins and libraries.

[0029] Preferably, the document and support unit include:

[0030] User manual, which provides guidance and operating instructions for the system;

[0031] API reference, which provides the system's API interface and function descriptions;

[0032] Technical support, providing technical support and problem-solving services.

[0033] Beneficial Effects

[0034] The present invention provides a cross-platform UI automation system based on a graphical interface. Compared with the prior art, it has the following beneficial effects:

[0035] 1. The UI element recognition module uses image recognition, control tree traversal and deep learning algorithms to identify UI elements with high precision, including advanced attributes such as position, size, type, text content, etc., providing a basis for precise operations; the event processing module and input simulation module support a variety of event processing and input simulation methods, adapting to different application scenarios and devices, and improving the flexibility and accuracy of automated operations.

[0036] 2. The script engine unit supports scripts written in multiple programming languages, and provides script parsers, execution environments, and debugging tools to ensure the correctness and executability of scripts, while facilitating developers to quickly locate and solve problems; the graphical interface tool unit provides script editors, control selectors, and log viewers, integrating version control systems, real-time previews, and multi-dimensional control filtering functions, which improves the efficiency of script writing, control selection, and log management.

[0037] 3. The extension plug-in and library unit provides control library, script library and custom plug-in functions, continuously updates the control library to support newly emerging UI controls and interaction modes, provides script templates for commonly used automation tasks, and allows developers to customize plug-ins and libraries, enhancing the scalability and flexibility of the system; the documentation and support unit provides user manuals, API references and technical support services to help developers get started quickly, consult API documents and solve problems. At the same time, it establishes an official community forum and plug-in market to promote communication and collaboration among developers and enhance the ecological vitality of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 is a system block diagram of the present invention;

[0039] Figure 2 A block diagram of a core framework unit in the system of the present invention;

[0040] Figure 3 is a block diagram of a script engine unit in the system of the present invention;

[0041] Figure 4 A block diagram of a graphical interface tool unit in the system of the present invention;

[0042] Figure 5 A block diagram of an expansion plug-in and a library unit in the system of the present invention;

[0043] Figure 6 A block diagram of the document and support units in the system of the present invention.

[0044] In the figure: 1. Core framework unit; 11. Cross-platform adaptation layer; 12. UI element recognition module; 13. Event processing module; 14. Input simulation module; 2. Script engine unit; 21. Script parser; 22. Script execution environment; 23. Script debugging tool; 3. Graphical interface tool unit; 31. Script editor; 32. Control selector; 33. Log viewer; 4. Extension plug-in and library unit; 41. Control library; 42. Script library; 43. Custom plug-in; 5. Documentation and support unit; 51. User manual; 52. API reference; 53. Technical support. DETAILED DESCRIPTION

[0045] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0046] See also Figure 1 - Figure 6 ,The present invention provides a technical solution: a cross-platform UI automation system based on a graphical interface, comprising a core framework unit 1, a script engine unit 2, a graphical interface tool unit 3, an extension plug-in and library unit 4 and a document and support unit 5;

[0047] Core framework unit 1, used to provide basic functions and interfaces, allowing developers to perform UI automation operations on different platforms;

[0048] The script engine unit 2 is used to allow developers to write automation scripts to define specific automation operation processes;

[0049] Graphical interface tool unit 3, used to provide an intuitive operation interface to create, edit and manage automated test scripts and visualization of test cases;

[0050] Extension plugins and libraries unit 4, used to provide additional functionality and support, as well as the implementation of custom script commands and tools;

[0051] Documentation and support unit 5 is used to provide system usage guides, API references, troubleshooting methods and user support services.

[0052] In this implementation, advanced UI element recognition technology is used in combination with deep learning algorithms to accurately capture and operate UI elements on the screen to ensure the accuracy of the test; at the same time, the system's built-in script engine supports multiple programming languages ​​and is equipped with powerful debugging tools, allowing developers to easily write, debug and execute automated scripts, greatly improving development efficiency; and the graphical interface tool provides an intuitive operating interface, reducing the difficulty of use, allowing developers to complete testing tasks more quickly.

[0053] Specifically, the core framework unit 1 includes:

[0054] The cross-platform adaptation layer 11 provides adaptation for UI element recognition, event processing, input simulation, etc. on different platforms, so that the system can achieve consistent automated operations on different operating systems;

[0055] The UI element recognition module 12 obtains the properties and status information of the UI elements through image recognition and control tree traversal, and provides basic data for subsequent automated operations;

[0056] The event processing module 13 monitors and processes UI events, triggers corresponding automation operations and executes corresponding script logic according to event types and event parameters;

[0057] The input simulation module 14 simulates the user's input behavior, such as keyboard input, mouse click and touch sliding.

[0058] In this embodiment, the cross-platform adaptation layer 11 goes deep into the bottom layer to handle the differences in UI presentation, event handling mechanism, input method, etc. between different operating systems (such as Windows, macOS, iOS, Android, etc.). By encapsulating the underlying API call and implementing the abstract interface, it is ensured that the upper logic can operate various platforms consistently, and the code path selection for a specific platform is realized by using the conditional compilation, dynamic link library loading and other technologies, while maintaining the unity of the core logic; in addition to the basic image recognition and control tree traversal technology, the UI element recognition module 12 also introduces deep learning algorithms (such as YOLO, SSD, etc.) for high-precision element recognition, especially in complex backgrounds or dynamically changing interfaces; not only the basic information such as position, size, type, etc. can be recognized, but also the advanced attributes such as text content, interactive state, hierarchical relationship of the element can be recognized, providing a basis for accurate operation; the event processing module 13 also supports more types of event processing, such as long press, double click, multi-finger touch, etc., to adapt to different application scenarios; in addition to the traditional keyboard input and mouse simulation, the input simulation module 14 also supports touch screen gesture simulation, voice input simulation, etc., to adapt to different devices and user habits.

[0059] Specifically, the script engine unit 2 includes:

[0060] The script parser 21 parses the automation script written by the developer and converts it into instructions that can be executed by the system, ensuring the correctness and executability of the script and providing basic instructions for the automation operation;

[0061] The script execution environment 22 provides an isolated execution environment for executing automated scripts, protecting the system core code from script errors, and providing the resources and interfaces required for script execution.

[0062] In this embodiment, the script parser 21 supports scripts written in multiple programming languages ​​(such as Python, JavaScript), and converts the scripts into internal instruction sets by defining grammar rules and parsers. The scripts are syntax checked and error prompts are given during the parsing stage to improve the efficiency and accuracy of script writing. The script execution environment 22 uses sandbox technology or containerization technology to provide an independent execution environment for each script to avoid mutual interference between scripts, and then dynamically allocates and recycles the resources required for script execution (such as memory and CPU time) to ensure the effective use of system resources.

[0063] Specifically, the script engine unit 2 also includes a script debugging tool 23 and is used to provide debugging functions such as breakpoints, single-step execution, and variable viewing, so as to quickly locate and solve problems when writing and debugging automation scripts.

[0064] In this embodiment, 23 provides a visual interface to display the variable status, call stack and other information during the script execution process, so that developers can intuitively understand the script behavior; and supports remote connection to devices or simulators for debugging, which is convenient for developers to test scripts in different environments.

[0065] Specifically, the graphical interface tool unit 3 includes:

[0066] Script Editor 31, provides code editing functions, supports syntax highlighting and automatic completion;

[0067] The control selector 32 provides a visual control selection function to quickly locate and select UI elements on the screen.

[0068] In this embodiment, the context-aware intelligent code completion in the script editor 31 improves writing efficiency, and integrates a version control system (such as Git) to support version control and collaborative development of scripts; the control selector 32 previews the selected UI elements in the editor in real time, which is convenient for developers to confirm intuitively, and provides the function of filtering controls by multiple dimensions such as attributes, levels, and types to improve selection accuracy.

[0069] Specifically, the graphical interface tool unit 3 further includes a log viewer 33 for displaying the execution log and error information of the automation script.

[0070] In this embodiment, the log viewer 33 supports log level settings (such as INFO, WARNING, ERROR), which is convenient for developers to quickly locate problems, and provides a log export function, which is convenient for developers to use logs for subsequent analysis or reporting.

[0071] Specifically, the extension plug-in and library unit 4 includes:

[0072] Control library 41, provides support and recognition capabilities for common UI controls, and expands the control recognition range of the system;

[0073] Script library 42, providing commonly used automation scripts and function libraries;

[0074] Custom plugins 43, allowing development of custom plugins and libraries.

[0075] In this embodiment, the control library 41 is continuously updated to support newly emerging UI controls and interaction modes, and an interface is provided to allow developers to write custom recognition logic for specific applications or controls; the script library 42 provides script templates for a variety of commonly used automation tasks, such as login, data entry, page navigation, etc., and supports script sharing and reuse mechanisms to promote knowledge sharing and collaboration within the community; the custom plug-in 43 provides detailed plug-in development documents and sample codes to lower the threshold for plug-in development, and a plug-in market can be established to allow developers to upload and download plug-ins to enhance the ecological vitality of the system.

[0076] Specifically, Documentation and Support Unit 5 includes:

[0077] User manual 51, providing guidance and operating instructions for the system;

[0078] API reference 52, providing the system API interface and function description;

[0079] Technical Support 53 provides technical support and problem-solving services.

[0080] In this embodiment, the user manual 51 provides a step-by-step tutorial from installation to use to help novices get started quickly, and shares best practice cases in the industry to improve developers' automated testing capabilities; the API reference 52 provides online API documentation for developers to access at any time, and records the API change history to help developers understand the differences between versions; the technical support 53 establishes an official community forum to encourage developers to exchange ideas and share experiences, and can provide paid professional support services, including quick response, customized solutions, etc.

[0081] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0082] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A cross-platform UI automation system based on a graphical interface, featuring: It includes a core framework unit (1), a script engine unit (2), a graphical interface tool unit (3), an extension plug-in and library unit (4) and a document and support unit (5); The core framework unit (1) is used to provide basic functions and interfaces, allowing developers to perform UI automation operations on different platforms; A script engine unit (2), used to allow developers to write automation scripts to define specific automation operation processes; A graphical interface tool unit (3), used to provide an intuitive operation interface for creating, editing and managing automated test scripts and visualization of test cases; Extension plugins and library units (4) for providing additional functionality and support, as well as implementation of custom script commands and tools; Documentation and support unit (5), used to provide system usage guides, API references, troubleshooting methods and user support services.

2. The cross-platform UI automation system based on a graphical interface according to claim 1, characterized in that: The core framework unit (1) comprises: The cross-platform adaptation layer (11) provides adaptation for UI element recognition, event processing, input simulation, etc. on different platforms, so that the system can achieve consistent automated operations on different operating systems; The UI element recognition module (12) obtains the attributes and status information of the UI elements through image recognition and control tree traversal, and provides basic data for subsequent automated operations; An event processing module (13) monitors and processes UI events, triggers corresponding automated operations and executes corresponding script logic according to event types and event parameters; The input simulation module (14) simulates the user's input behavior, such as keyboard input, mouse click and touch sliding.

3. The cross-platform UI automation system based on a graphical interface according to claim 1, characterized in that: The script engine unit (2) comprises: The script parser (21) parses the automation script written by the developer and converts it into instructions that can be executed by the system, ensuring the correctness and executability of the script and providing basic instructions for the automation operation; The script execution environment (22) provides an isolated execution environment for executing automated scripts, protecting the system core code from script errors, and providing the resources and interfaces required for script execution.

4. The cross-platform UI automation system based on a graphical interface according to claim 1, characterized in that: The script engine unit (2) also includes a script debugging tool (23) for providing debugging functions such as breakpoints, single-step execution, and variable viewing, so as to quickly locate and solve problems when writing and debugging automation scripts.

5. The cross-platform UI automation system based on a graphical interface according to claim 1, characterized in that: The graphical interface tool unit (3) comprises: Script Editor (31), which provides code editing functions and supports syntax highlighting and auto-completion features; The control selector (32) provides a visual control selection function to quickly locate and select UI elements on the screen.

6. The cross-platform UI automation system based on a graphical interface according to claim 1, characterized in that: The graphical interface tool unit (3) also includes a log viewer (33) and is used to display the execution log and error information of the automation script.

7. The cross-platform UI automation system based on a graphical interface according to claim 1, characterized in that: The extension plug-in and library unit (4) comprises: Control library (41), providing support and recognition capabilities for common UI controls, expanding the control recognition range of the system; Script library (42), providing commonly used automation scripts and function libraries; Custom plugins (43), allowing development of custom plugins and libraries.

8. The cross-platform UI automation system based on a graphical interface according to claim 1, characterized in that: The document and support unit (5) comprises: User manual (51), which provides guidance and operating instructions for the system; API reference (52), providing the system's API interface and function descriptions; Technical Support (53), providing technical support and problem-solving services.

Images