An embedded system UI automatic test method and system based on ARM architecture

Abstract

This invention discloses an automated testing method and system for embedded system UI based on ARM architecture. The system includes: a keyboard and mouse chip driver, providing an interface to the application layer for reading physical keyboard and mouse events; a virtual keyboard and mouse driver, simulating physical input devices; a recording module, which reads keyboard and mouse event data by calling the keyboard and mouse chip driver interface and saves the event data and time information as a script file; and a playback module, which reads the script file and writes the event data to the virtual keyboard and mouse driver according to the time information to simulate real keyboard and mouse operations. This invention achieves non-intrusive, high-fidelity automated testing of embedded system UIs by recording and playing back events at the driver layer, without requiring modification of the application's source code, effectively solving the problems of low testing efficiency and poor stability in existing technologies.

Description

Technical Field

[0001] This invention relates to the field of embedded system testing technology, and more specifically, to an automated testing method and system for the user interface (UI) of an embedded system based on ARM architecture. Background Technology

[0002] Traditional embedded system UI testing relies heavily on manual operation, which is inefficient, involves a large amount of repetitive work, is prone to errors, and is difficult to automate in continuous integration / continuous deployment (CI / CD) processes. Existing automated testing solutions also have the following limitations: Image recognition-based methods compare screen images captured by a camera. However, this method is highly susceptible to variations in resolution, lighting, and even slight changes in UI elements, resulting in high computational costs and making it unsuitable for resource-constrained embedded environments.

[0003] The method of directly calling the Qt API via scripts requires direct access to the application's source code or internal APIs. This is impractical in many cases, especially when the tester is not the application developer (e.g., a third-party tester or system integrator). This method is also tightly coupled to a specific UI framework (such as Qt) and lacks versatility.

[0004] System-level keyboard and mouse emulation: Some existing tools (such as xdotool for Linux) can simulate keyboard and mouse input, but they usually operate at a high level of the operating system and cannot reliably handle embedded environments that interact closely with hardware, nor can they record accurate input sequences with timing information from real hardware.

[0005] Therefore, there is an urgent need in this field for a UI automation testing solution that can work stably in an embedded environment without modifying the application under test, without relying on image recognition. Summary of the Invention

[0006] This invention aims to address the shortcomings of existing technologies by providing an automated testing method and system for embedded system UI based on ARM architecture. This method intercepts real keyboard and mouse events at the kernel driver layer and replays them to the virtual keyboard and mouse driver, achieving precise recording and playback of user interface (UI) operations, thereby achieving efficient, stable, and non-intrusive automated testing.

[0007] To achieve the above objectives, the present invention adopts the following technical solution.

[0008] This invention provides an automated UI testing system for embedded systems based on ARM architecture, comprising: Keyboard and mouse chip drivers are used to control keyboard and mouse hardware and provide the application layer with an interface to read input event data originating from the physical keyboard and mouse. The recording module reads input event data from the physical keyboard and mouse by calling the application layer interface provided by the keyboard and mouse chip driver, and saves the event data and time information together as a script file; The playback module reads the script file and writes the event data to the input interface of the virtual keyboard and mouse driver based on the time information. Virtual keyboard and mouse drivers are used to receive data written by the application layer and simulate physical keyboards and mice, thereby triggering automated operations of the embedded system's user interface (UI).

[0009] In this invention, the keyboard and mouse chip driver provides a data reading interface to the application layer through the device file node.

[0010] In this invention, the embedded system uses an ARM processor as the core computing unit, and the physical keyboard and mouse are directly connected to the keyboard and mouse chip controlled by the ARM.

[0011] In this invention, input event data includes keyboard key values, press / release states, and / or mouse coordinates, click states, and scroll states.

[0012] In this invention, the virtual keyboard and mouse driver is implemented based on the uinput framework of the Linux operating system.

[0013] This invention provides an automated UI testing method for ARM-based embedded systems, which, based on the aforementioned system implementation, includes: The keyboard and mouse chip driver controls the keyboard and mouse hardware and provides the application layer with an interface to read input event data originating from the physical keyboard and mouse. The recording module reads input event data from the physical keyboard and mouse by calling the application layer interface provided by the keyboard and mouse chip driver, and saves the event data and time information together as a script file; The recording module reads the script file and, based on the time information, writes the event data to the input interface of a virtual keyboard and mouse driver. The playback module's virtual keyboard and mouse driver receives data written by the application layer and simulates a physical keyboard and mouse, thereby triggering automated operations on the user interface.

[0014] Furthermore, the present invention provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the aforementioned embedded system UI automated testing method.

[0015] Compared with the prior art, the present invention has the following beneficial effects: 1. Non-intrusive: No need to modify the source code of the Qt application under test, achieving decoupling from the application and strong universality.

[0016] 2. High fidelity and reliability: Event recording and playback at the driver layer accurately reproduce user operations, including complex key combinations, mouse movements, and precise timing. Compared to application-layer simulation, it triggers system events at a lower level and more reliably, overcoming the instability of higher-level simulation.

[0017] 3. High resource efficiency: Compared with image recognition-based solutions, this solution has extremely low computational overhead, making it very suitable for resource-constrained embedded environments.

[0018] 4. Simple architecture and low cost: It eliminates the FPGA and adopts a pure ARM architecture. It uses software drivers to implement core functions, which reduces system complexity and hardware costs and makes it easier to promote.

[0019] 5. Easy to use: It provides a simple operation similar to "macro recording", allowing testers to create automated test scripts without writing code, which greatly reduces the barrier to entry. Attached Figure Description

[0020] Figure 1 This is a diagram of the overall system architecture.

[0021] Figure 2 This is a flowchart of the recording process.

[0022] Figure 3 This is a flowchart of the playback process.

[0023] Figure 4 This is a sample code snippet (conceptual) for the recording module and the postback module in the system of this invention. Detailed Implementation

[0024] The technical solution of the present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0025] Figure 1 This is a diagram of the overall system architecture. (For example...) Figure 1 As shown, the ARM-based embedded system UI automated testing system of the present invention includes: 1. Hardware Layer: An embedded hardware platform based on an ARM processor. The physical keyboard and mouse are directly connected to the keyboard and mouse chip controlled by the ARM processor.

[0026] 2. Driver layer: Located in the operating system kernel, including: Keyboard and mouse chip driver: Used to control the keyboard and mouse chip and provide an interface to the application layer for reading raw keyboard and mouse events. The interface of the keyboard and mouse chip driver describes how the driver exposes data to the application layer. For example, it can be provided to the application layer for reading through a device file (such as / dev / input / eventX) or a custom ioctl interface.

[0027] Standard keyboard and mouse driver: The default driver in the operating system used to handle keyboard and mouse input events.

[0028] Virtual keyboard and mouse driver: This invention is a custom-developed driver program that receives input data and simulates it as physical keyboard and mouse events, reporting them to the operating system kernel to trigger standard keyboard and mouse drivers and upper-layer Qt events. The virtual keyboard and mouse driver can be implemented based on the Linux uinput framework. Details are provided on how to create and configure a virtual input device, and how to "write" data to the device to simulate input (e.g., by writing struct input_event to / dev / uinput).

[0029] Application layer: Runs the Qt user interface application to be tested. Figure 4 This is a sample code snippet (conceptual) for the recording module and the postback module in the system of this invention.

[0030] Recording Module: As the core of this invention, this module runs at the application layer. It reads raw keyboard and mouse event data by calling the interface provided by the keyboard and mouse chip driver. This module records the raw input data stream, including key codes, mouse coordinates, mouse clicks, and timestamps, and serializes and saves it as a recording script file. The recording module describes how to open the device file of the keyboard and mouse chip driver, read the `struct input_event` in a loop, add a timestamp, and serialize it to the file. The data structure of the recording script file, i.e., the data format, can be a binary file storing a series of records, each containing: timestamp (relative to the start time), event type, event code, and event value.

[0031] Playback Module: As another core component of this invention, this module also operates at the application layer. It reads the recording script file and writes the recorded input data to the input interface of the virtual keyboard and mouse driver. The virtual keyboard and mouse driver then simulates this data as real hardware input and reports it to the system, ultimately driving the Qt application to perform the exact same operations as when it was recorded. The playback module describes how to open the virtual keyboard and mouse driver (e.g., / dev / uinput), read events from the script file, use usleep or nanosleep to wait for precise time intervals based on timestamps, and then write the events to the virtual driver.

[0032] Furthermore, the embedded system UI automated testing method based on the above system includes the following steps: 1) Recording process ( Figure 2 ): Step S1: Start the recording module; Step S2: The user operates the physical keyboard and mouse; Step S3: The keyboard and mouse chip driver captures input events from the hardware and provides them to the application layer through the interface; Step S4: The recording module reads keyboard and mouse event data through the driver interface and saves the input events and related metadata (such as precise timestamps) to the recording file; Step S5: Stop recording and generate a complete recording script file.

[0033] II) Playback process ( Figure 3 ): Step P1: Start the playback module and load the specified recording script file; Step P2: The playback module parses the script file and reads the input event data according to the recorded time sequence; Step P3: The playback module writes the input data to the input interface of the virtual keyboard and mouse driver; Step P4: The virtual keyboard and mouse driver simulates the received data as physical keyboard and mouse input and reports it to the operating system's standard keyboard and mouse driver; Step P5: The operating system generates corresponding Qt input events, driving the application to automatically execute the recorded operation sequence; Step P6: Testers observe the application's running status and results to complete automated testing.

Claims

1. An automated testing system for embedded system UI based on ARM architecture, characterized in that, include: Keyboard and mouse chip drivers are used to control keyboard and mouse hardware and provide the application layer with an interface to read input event data originating from the physical keyboard and mouse. The recording module reads input event data from the physical keyboard and mouse by calling the application layer interface provided by the keyboard and mouse chip driver, and saves the event data and time information together as a script file; The playback module reads the script file and writes the event data to the input interface of the virtual keyboard and mouse driver based on the time information. Virtual keyboard and mouse drivers are used to receive data written by the application layer and simulate physical keyboards and mice, thereby triggering automated operations of the embedded system's user interface (UI).

2. The embedded system UI automated testing system as described in claim 1, characterized in that, Keyboard and mouse chip drivers provide data reading interfaces to the application layer through device file nodes.

3. The embedded system UI automated testing system as described in claim 1, characterized in that, The embedded system uses an ARM processor as the core computing unit, and the physical keyboard and mouse are directly connected to the keyboard and mouse chip controlled by the ARM.

4. The embedded system UI automated testing system as described in claim 1, characterized in that, Input event data includes keyboard key values, press / release states, and / or mouse coordinates, click states, and scroll states.

5. The embedded system UI automated testing system as described in claim 1, characterized in that, The virtual keyboard and mouse driver is implemented based on the uinput framework of the Linux operating system.

6. An automated testing method for the UI of an embedded system based on ARM architecture, characterized in that, Its implementation, based on the system described in claim 1, includes the following steps: The keyboard and mouse chip driver controls the keyboard and mouse hardware and provides the application layer with an interface to read input event data originating from the physical keyboard and mouse. The recording module reads input event data from the physical keyboard and mouse by calling the application layer interface provided by the keyboard and mouse chip driver, and saves the event data and time information together as a script file; The playback module reads the script file and writes the event data to the input interface of the virtual keyboard and mouse driver based on the time information. The virtual keyboard and mouse driver receives data written by the application layer and simulates a physical keyboard and mouse, thereby triggering automated operations of the embedded system's user interface (UI).

7. The embedded system UI automated testing method according to claim 6, characterized in that, The script file is a binary file that stores a series of records, each containing: timestamp, event type, event code, and event value.

8. The embedded system UI automated testing method according to claim 6, characterized in that, The virtual keyboard and mouse driver receives data written by the application layer and simulates a physical keyboard and mouse, thereby triggering automated operations of the embedded system's user interface (UI). The specific method is as follows: (1) The virtual keyboard and mouse driver simulates the received data as physical keyboard and mouse input and reports it to the standard keyboard and mouse driver of the operating system; (2) The operating system generates corresponding Qt input events, driving the application to automatically execute the recorded operation sequence; (3) Testers observe the running status and results of the application and complete automated testing.

9. A computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the method as described in any one of claims 6-8.

Images