Machine vision-based industrial ui automation testing method and system

Abstract

The application discloses a kind of industrial UI automation test method and system based on machine vision, belongs to industrial automation test technical field, comprising: using three-axis holder bearing stylus and image acquisition device, through first controller generates operation instruction to drive three-axis holder to execute click and sliding operation on the display device to be measured, while image acquisition device moves with stylus and collects interface image sequence;Second controller receives sampling trigger signal, and pre-processes interface image sequence, and carries out interface state recognition by the image analysis model of fusion CNN feature extraction and OCR text recognition;First controller determines to continue testing or records abnormal log according to the recognition result and generates test report.The application does not need to rely on the communication interface or preset test firmware of measured equipment, realizes generalization UI automation test by machine vision, significantly improves test efficiency and quality assurance ability.

Description

Technical Field

[0001] This invention belongs to the field of industrial automation testing technology, and in particular, it is a novel automated testing method and system for human-machine interface (HMI) that integrates machine vision and electromechanical control. Background Technology

[0002] In the research and development and production testing of industrial equipment (especially small and medium-sized industrial terminals, PLC control panels, embedded human-machine interface devices, etc.), traditional automated testing solutions mainly rely on two technical approaches: hardware interface communication testing and customized software-driven testing. The former verifies functional logic by directly reading data from the device's internal bus (such as Modbus, CAN) or I / O interface, while the latter requires pre-installing test firmware on the device or deep integration with the operating system, simulating user operations through code-level calls. Typical application scenarios include teach pendant testing of industrial robots and verification of display and parameter settings of intelligent instruments.

[0003] Taking a certain type of industrial HMI (Human-Machine Interface) testing system as an example, its structure typically includes: ① an industrial control computer (running test scripts); ② a dedicated communication module (such as an RS485 / Ethernet adapter board); and ③ a device control interface (such as GPIO trigger buttons and analog input modules). The testing principle is as follows: the industrial control computer sends commands to the device via a communication protocol (such as "set the temperature parameter to 50℃"), while simultaneously monitoring the status code or data returned by the device. If the data matches expectations, the test is considered passed. For low-cost devices without network connectivity, some solutions employ physical probe contact testing, connecting the device's circuit nodes via a bed of needles to directly detect changes in signal levels.

[0004] However, the aforementioned prior art has the following technical problems: First, it lacks versatility. Existing solutions heavily rely on the communication interface and protocol stack of the device under test. For low-cost industrial equipment that lacks networking capabilities or has not opened its underlying protocols, automated testing cannot be directly implemented. Specialized testing fixtures need to be developed, which leads to longer testing cycles and increased costs.

[0005] Second, defect identification relies on manual visual inspection. Existing automated solutions are ineffective in detecting abnormalities in the UI of industrial equipment (such as missing characters, disordered layout, and incorrect display of fault prompts). They mainly rely on manual observation by testers, resulting in low efficiency and a high rate of missed detections, which is particularly prominent in large-scale production line testing scenarios.

[0006] Third, test script development is costly. Traditional automated testing requires pre-writing detailed test scripts, hard-coding operation paths, click coordinates, waiting times, etc. When the UI version of the device under test is iterated or the operation process changes, the workload of script maintenance is huge.

[0007] Fourth, fault tracing is difficult. When an anomaly is detected, existing solutions typically only record communication timeouts or incorrect return values, lacking a record of the correlation between the UI screen and operation sequence before and after the fault. This makes it difficult to reproduce the problem, and developers find it hard to quickly pinpoint whether the issue is a UI rendering problem, a logic error, or a hardware failure. Summary of the Invention

[0008] To address the aforementioned technical problems, this invention aims to provide an automated testing method and system for industrial UI based on machine vision. By replacing hardware interface dependence with image acquisition and visual recognition, it achieves lightweight, universal, and intelligent automated testing of industrial equipment UI.

[0009] The technical solution of this invention is: In a first aspect, the present invention provides an automated testing method for industrial UI based on machine vision, comprising: S1. A three-axis gimbal is connected to a stylus to perform simulated user operations on the display device under test. The simulated user operations include click and swipe actions. An image acquisition device is fixed to the three-axis gimbal and moves with the stylus to acquire a sequence of interface images. The sequence of interface images covers the changes in the interface before and after the simulated user operation. S2. The first controller MCU generates operation instructions and sends sampling trigger signals to the second controller ARM through the wireless communication module; S3. The second controller receives the sampling trigger signal, acquires the interface image sequence for preprocessing, and performs interface state recognition through an image analysis model. The image analysis model integrates image feature extraction and text recognition, and outputs normal or abnormal interface classification results. S4. The second controller feeds back the interface state recognition result to the first controller. The first controller decides whether to continue to execute the next simulated user operation or record the exception log based on the interface state recognition result, and generates a test report after the test is completed.

[0010] Furthermore, the three-axis gimbal used in S1 to carry the stylus and perform simulated user operations on the display device under test includes: using a first controller to parse the pre-stored operation script to generate control signals, driving the motor of the three-axis gimbal to drive the stylus on the display device under test to simulate user operations; the pre-stored operation script comes from manually recorded operation sequences or random operation sequences.

[0011] Furthermore, the manually recorded operation sequence is completed by the user manually controlling the stylus to complete the operation trajectory. The first controller records the number of pulses and time intervals of each axis motor of the three-axis gimbal and stores them as the pre-stored operation script. The random operation sequence is generated by the first controller with coordinate offset and waiting time. The coordinate offset corresponds to the click action and the swipe action.

[0012] Furthermore, S2 includes: the first controller and the second controller establish a communication link through a wireless communication module and synchronize a time reference; a three-axis gimbal is used to move the image acquisition device to the initial position, and calibration is completed through a pre-stored interface template, wherein the pre-stored interface template corresponds to the initial interface of the display device under test.

[0013] Furthermore, after receiving the sampling trigger signal, the second controller in S3 controls the image acquisition device to continuously acquire the interface image sequence, preprocesses the interface image sequence, and then inputs it into the image analysis model. The image preprocessing includes grayscale processing and noise reduction processing.

[0014] Furthermore, the image analysis model described in S3 performs the following steps: using a CNN unit to extract the position, shape, and color features of the interface elements in the interface image sequence; using an OCR unit to recognize the text content of the interface image sequence and form a feature vector; concatenating the position, shape, and color features of the interface elements and the feature vector to output a normal or abnormal interface classification result.

[0015] Furthermore, in the image analysis model, the image branch is flattened to the first feature layer after being processed by convolutional and pooling layers; the text branch is used to encode preset interface keywords to the second feature layer; the first feature layer and the second feature layer are fused, and the normal or abnormal interface classification result is output through a fully connected layer.

[0016] Furthermore, in S4, if the interface state recognition result is an abnormal interface classification result, the first controller controls the display device to update the abnormal detection information; the second controller extracts the interface image sequence frame and operation timestamp associated with the abnormal interface classification result, and packages and stores them as an abnormal log.

[0017] Furthermore, after the test in S4 is completed, the first controller controls the three-axis gimbal to reset to the initial position; the second controller summarizes the abnormal logs and operation command sequences to generate a test report, which includes the interface image sequence frames and abnormal distribution information.

[0018] Secondly, the present invention provides an automated testing system for industrial UI based on machine vision, comprising: A three-axis gimbal is mounted on which a stylus and an image acquisition device are fixed. The three-axis gimbal is used to respond to control signals to drive the stylus to perform simulated user operations on the display device under test, and to move the image acquisition device with the stylus to acquire a sequence of interface images covering the changes in the interface before and after the simulated user operations; wherein, the simulated user operations include click actions and swipe actions. The first controller is an MCU, which is configured to generate operation commands and send sampling trigger signals through a wireless communication module. The second controller, using an ARM architecture, is connected to the first controller via a wireless communication module and is configured as follows: Receive and respond to the sampling trigger signal, acquire the interface image sequence acquired by the image acquisition device, and preprocess the interface image sequence. The image analysis model is used to identify the interface state of the preprocessed interface image sequence. The image analysis model integrates image feature extraction and text recognition, and outputs the classification result of normal or abnormal interface. The first controller is further configured to: determine whether to continue executing the next simulated user operation or record an exception log based on the interface state recognition result, and generate a test report after the test is completed.

[0019] Compared with the prior art, the present invention has the following advantages: 1. Significantly Improved Testing Efficiency: This invention utilizes a three-axis gimbal equipped with a stylus to simulate user operations, combined with an image acquisition device to capture real-time interface image sequences, achieving closed-loop automated testing of "operation-acquisition-recognition". Taking the screen of an electric vehicle charging station terminal as an example, the testing cycle for a single device is reduced from 10 minutes in the traditional solution to 128 seconds, improving testing efficiency by approximately 78.7%. Simultaneously, the system supports clustered parallel testing; a single first controller can connect to multiple test units simultaneously via a wireless communication module, theoretically supporting parallel testing of 254 devices, meeting the needs of large-scale production testing.

[0020] 2. Significantly Improved UI Defect Detection Rate: This invention employs an image analysis model that integrates CNN feature extraction and OCR text recognition. It can automatically distinguish between fault warning pages (such as "sensor disconnection" and "parameter exceeding limits") and display defect pages (such as missing characters and layout errors), achieving high-precision identification of UI interface anomalies. Actual testing, using electric vehicle charging piles as the test object, shows that the traditional manual testing method, with one test engineer conducting three rounds of testing, achieves a UI fault detection rate of approximately 65%. In contrast, this invention achieves a UI fault detection rate of 87%, with a false alarm rate controlled below 3%, significantly improving product quality assurance capabilities.

[0021] 3. Significantly Reduced Testing Costs: This invention eliminates the need for the device under test to open its communication protocols or have pre-installed test firmware. The entire testing process can be completed solely through machine vision, reducing reliance on dedicated testing fixtures. Compared to existing technologies, it reduces testing fixture development costs by 90% and product testing manpower costs by 80%. Furthermore, the operation recording and playback function supports "one-time recording, multiple reproductions," shortening test script development and maintenance time, and reducing the overall product development cycle by approximately 30%.

[0022] 4. Short investment payback period: This invention is implemented in a modular manner, with controllable hardware costs, and is suitable for production line testing scenarios of various industrial equipment. Calculations show that the equipment investment payback period is less than 6 months, demonstrating good economic benefits and significant value for widespread application.

[0023] 5. Enhanced Fault Tracing Capability: When an abnormal interface is detected, this invention automatically captures the associated interface image sequence frames and operation timestamps to form a "operation-screen-result" associated log. This mechanism helps developers quickly distinguish between UI rendering problems, logic errors, or hardware failures, shortening the problem localization and repair cycle.

[0024] Other features and advantages of the present invention will be described in detail in the following detailed description section. Attached Figure Description

[0025] The above and other objects, features and advantages of the present invention will become more apparent from the more detailed description of exemplary embodiments of the invention in conjunction with the accompanying drawings, wherein the same reference numerals generally represent the same components in the exemplary embodiments of the invention.

[0026] Figure 1 The flowchart of the machine vision-based industrial UI automated testing method of the present invention is shown.

[0027] Figure 2 The diagram shows the architecture of the machine vision-based industrial UI automated testing system of the present invention. Detailed Implementation

[0028] Preferred embodiments of the invention will now be described in more detail with reference to the accompanying drawings. While preferred embodiments of the invention are shown in the drawings, it should be understood that the invention can be implemented in various forms and should not be limited to the embodiments set forth herein.

[0029] like Figure 1 , 2 As shown, this invention provides an automated testing method for industrial UI based on machine vision, comprising: S1. A three-axis gimbal is connected to a stylus to perform simulated user operations on the display device under test. The simulated user operations include click and swipe actions. An image acquisition device is fixed to the three-axis gimbal and moves with the stylus to acquire a sequence of interface images. The sequence of interface images covers the changes in the interface before and after the simulated user operation. S1. A three-axis gimbal is connected to a stylus to perform simulated user operations on the display device under test. The simulated user operations include click and swipe actions. An image acquisition device is fixed to the three-axis gimbal and moves with the stylus to acquire a sequence of interface images. The sequence of interface images covers the changes in the interface before and after the simulated user operation.

[0030] Specifically, this includes: S11, initializing the three-axis gimbal and connecting the stylus; the three-axis gimbal consists of X-axis, Y-axis, and Z-axis stepper motors, each with a step angle of 1.8 degrees and a torque of not less than 0.5 Nm, ensuring a positioning accuracy of ±0.1 mm. The stylus is made of conductive rubber, 5 mm in diameter, and rigidly connected to the end of the gimbal's Z-axis. During initialization, the gimbal moves to a preset initial position, such as 10 cm directly above the screen of the device under test. S12, executing simulated user operations; simulated user operations generate random or recorded sequences through the first controller, driving the gimbal to perform click actions, such as the stylus quickly descending to contact a specific coordinate on the screen and then lifting; sliding actions involve the stylus moving continuously along the X-axis or Y-axis, such as sliding 50 pixels from coordinates (200, 300). S13, fixing the image acquisition device and acquiring sequences; the image acquisition device, such as an OV7725 camera with a resolution of 640 x 480 and a frame rate of 30 frames per second, is fixed to the end of the gimbal's Z-axis and moves synchronously with the stylus. When acquiring the sequence of interface images, for each operation, the system acquires 4 frames: 1 frame before the operation, 1 frame during the operation, and 2 frames after the operation, covering the interface changes within 100 milliseconds.

[0031] In one embodiment, for testing an industrial robot teach pendant, in step S1, the three-axis gimbal is first calibrated to the center of the teach pendant screen. A simulated operation of clicking the "Start" button is then performed. The stylus descends to make contact and then lifts up, while the camera captures the sequence of events before and after the button is pressed, changing the interface from a static menu to a dynamic progress bar. This method ensures that the test covers operational responses and improves fault detection accuracy.

[0032] For example, in the testing of medical equipment operating terminals, the gimbal is connected to the touch pen to slide and adjust the parameter bar, and the camera follows to collect the sequence and record the changes in values ​​before and after the slide. This is helpful in identifying delay response defects and shortening the maintenance cycle.

[0033] S2. The first controller MCU generates operation instructions and sends sampling trigger signals to the second controller ARM through the wireless communication module; the first controller, such as STM32F407, after generating operation instructions, sends sampling trigger signals through the ESP8266 wireless module in accordance with the 802.11 protocol, with a transmission distance of not less than 10 meters.

[0034] S3. The second controller receives the sampling trigger signal, acquires the interface image sequence for preprocessing, and performs interface state recognition through an image analysis model. The image analysis model integrates image feature extraction and text recognition, and outputs normal or abnormal interface classification results. Specifically, the process includes: S31, receiving a sampling trigger signal and acquiring an image sequence. A second controller, such as an RK3568, immediately acquires a pre-stored interface image sequence from the camera upon receiving the signal. S32, performing image preprocessing. The sequence is converted to grayscale, transforming the RGB image into a single-channel grayscale image with pixel values ​​ranging from 0 to 255; then, Gaussian filtering is applied for noise reduction, with a 3x3 kernel size and a standard deviation of 1.0 to reduce noise interference. S33, performing recognition using an image analysis model. The image analysis model employs a convolutional neural network (CNN) combined with optical character recognition (OCR). First, the CNN extracts image features. This process involves inputting a 360x640x3 dimensional image, passing it through a 3x3 convolutional kernel, a 32-bit filter, and ReLU activation, outputting a 358x638x32 dimensional image. Then, a 2x2 max-pooling window is applied, outputting a 179x319x32 dimensional image. Next, a second convolutional layer with a 3x3 kernel, a 64-bit filter, and ReLU activation yields a 177x317x64 dimensional image. A second max-pooling layer outputs an 88x158x64 dimensional image, which is then flattened into a one-dimensional 888832 dimensional image. A fully connected layer with 128 neurons and ReLU activation outputs a 128-dimensional feature. Simultaneously, the text recognition branch inputs a 64-dimensional keyword binary encoding vector, passes through a fully connected layer with 128 neurons and ReLU activation, and outputs a 128-dimensional feature. The two branches are concatenated to form a 256-dimensional feature, which is then passed through a fully connected layer with 128 neurons and ReLU activation, and finally through a 2-neuron Softmax layer to output a normal or abnormal classification. S34: Output the classification result. Based on the splicing features, the model calculates the probability. A value above 0.5 is considered normal, otherwise it is considered abnormal.

[0035] In one embodiment, step S3 is applied to the PLC control panel test. After receiving the signal, the preprocessing sequence captures the parameter setting interface, the model extracts the button shape features and recognizes the text "Parameter saved successfully", and outputs a normal result. This is beneficial for automating the verification of operation logic and reducing manual intervention.

[0036] For example, in the testing of electric vehicle charging pile terminals, the preprocessed model is fused to extract color features and identify the text of "fault code E03", outputting anomaly classification, quickly locating and displaying defects, and improving the detection rate to 87%.

[0037] S4. The second controller feeds back the interface state recognition result to the first controller. The first controller decides whether to continue to execute the next simulated user operation or record the exception log based on the interface state recognition result, and generates a test report after the test is completed.

[0038] Specifically, this includes: S41, Feedback on identification results. The second controller feeds back the classification results and confidence levels to the first controller via a wireless module. S42, Decision-making based on results. If the results are normal, the first controller continues to the next operation; if abnormal, it logs the fault type, timestamp, and associated images. S43, Generation of test report. After the test is completed, the first controller summarizes the logs and images to generate a PDF report containing a fault heatmap.

[0039] For example, in industrial HMI testing, after the S4 step reports an abnormal result, the first controller logs and stops operating, generating a report that displays a high-frequency abnormal interface, which helps to shorten the problem repair cycle by 30%.

[0040] Secondly, the present invention provides an automated testing system for industrial UI based on machine vision, comprising: A three-axis gimbal is mounted on which a stylus and an image acquisition device are fixed. The three-axis gimbal is used to respond to control signals to drive the stylus to perform simulated user operations on the display device under test, and to move the image acquisition device with the stylus to acquire a sequence of interface images covering the changes in the interface before and after the simulated user operations; wherein, the simulated user operations include click actions and swipe actions. In one embodiment, a three-axis stepper motor gimbal (X / Y / Z axes, step angle 1.8°, torque ≥0.5 N·m) is used to carry the stylus and precisely control its position on the UI screen (positioning accuracy ±0.1 mm); the stylus, made of conductive rubber, is 5 mm in diameter and rigidly connected to the end of the gimbal to simulate human hand click / swipe operations. The image acquisition unit, an OV7725 camera module (640×480 resolution, 30fps frame rate, supports automatic exposure / white balance), is fixed to the Z-axis end of the gimbal, with the lens vertically downwards and aimed at the UI screen of the device under test.

[0041] The first controller is an MCU, which is configured to generate operation commands and send sampling trigger signals through a wireless communication module. In one embodiment, the 3-channel PWM output interface of the MCU board (such as STM32F407) amplifies the current through the A4988 driver chip to drive the motor to rotate; the image processing unit is an ARM board (such as RK3568) equipped with a Linux system, integrating a GPU (Mail-T860MP4) to accelerate neural network inference, and has a built-in multimodal model (CNN layer for interface element feature extraction, LSTM-OCR module for text recognition).

[0042] The second controller, using an ARM architecture, is connected to the first controller via a wireless communication module and is configured as follows: Receive and respond to the sampling trigger signal, acquire the interface image sequence acquired by the image acquisition device, and preprocess the interface image sequence. The image analysis model is used to identify the interface state of the preprocessed interface image sequence. The image analysis model integrates image feature extraction and text recognition, and outputs the classification result of normal or abnormal interface. The first controller is further configured to: determine whether to continue executing the next simulated user operation or record an exception log based on the interface state recognition result, and generate a test report after the test is completed.

[0043] In one embodiment, the ARM board (RK3568) communicates with the MCU board via UART or SPI, receives a sampling trigger command, calls OpenCV to process the image, runs the TensorFlow Lite model to complete UI classification, and sends the results back to the MCU via WiFi; it has built-in EMMC storage (capacity ≥16GB) for storing UI template libraries, fault screenshots and test reports.

[0044] The various embodiments of the present invention have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments.

Claims

1. A machine vision-based industrial Ul automation testing method, characterized by, include: S1. A three-axis gimbal is connected to a touch pen to perform simulated user operations on the display device under test. The simulated user operations include click actions and swipe actions. The image acquisition device is fixed to the three-axis gimbal and moves with the stylus to acquire a sequence of interface images. The sequence of interface images covers the changes in the interface before and after the simulated user operation. S2. The first controller generates operation instructions and sends sampling trigger signals to the second controller via the wireless communication module; S3. The second controller receives the sampling trigger signal, acquires the interface image sequence for preprocessing, and performs interface state recognition through an image analysis model. The image analysis model integrates image feature extraction and text recognition, and outputs normal or abnormal interface classification results. S4. The second controller feeds back the interface state recognition result to the first controller. The first controller decides whether to continue to execute the next simulated user operation or record the exception log based on the interface state recognition result, and generates a test report after the test is completed.

2. The method of claim 1, wherein In S1, the three-axis gimbal used to carry the stylus and perform simulated user operations on the display device under test includes: using a first controller to parse the pre-stored operation script to generate control signals, driving the motor of the three-axis gimbal to drive the stylus on the display device under test to simulate user operations; the pre-stored operation script comes from manually recorded operation sequences or random operation sequences.

3. The method of claim 2, wherein, The manually recorded operation sequence is completed by the user manually controlling the stylus to complete the operation trajectory. The first controller records the number of pulses and time intervals of the motors of each axis of the three-axis gimbal and stores them as the pre-stored operation script. The random operation sequence is generated by the first controller with coordinate offset and waiting time. The coordinate offset corresponds to the click action and the swipe action.

4. The method as described in claim 1, characterized in that S2 include: The first controller and the second controller establish a communication link through a wireless communication module and synchronize their time base. A three-axis gimbal is used to move the image acquisition device to its initial position, and calibration is completed using a pre-stored interface template, which corresponds to the initial interface of the display device under test.

5. The method of claim 1, wherein In S3, after receiving the sampling trigger signal, the second controller controls the image acquisition device to continuously acquire interface image sequences, preprocesses the interface image sequences, and then inputs them into the image analysis model. The image preprocessing includes grayscale processing and noise reduction processing.

6. The method of claim 1, wherein In S3, the image analysis model performs the following steps: CNN units are used to extract the position, shape, and color features of interface elements in the interface image sequence; The text content of the interface image sequence is recognized using an OCR unit to form a feature vector; By concatenating the position, shape, and color features of the interface elements, as well as the feature vector, the result of classifying the interface as normal or abnormal is output.

7. The method of claim 6, wherein, In the image analysis model, the image branch is processed by convolutional and pooling layers and then flattened to the first feature layer; the text branch is used to encode preset interface keywords to the second feature layer; the first feature layer and the second feature layer are fused, and the normal or abnormal interface classification result is output through a fully connected layer.

8. The method of claim 1, wherein In S4, if the interface state recognition result is an abnormal interface classification result, the first controller controls the display device to update the abnormal detection information; the second controller extracts the interface image sequence frame and operation timestamp associated with the abnormal interface classification result, and packages and stores them as an abnormal log.

9. The method of claim 1, wherein In S4, after the test is completed, the first controller controls the three-axis gimbal to reset to the initial position; the second controller summarizes the abnormal logs and operation command sequences to generate a test report, which includes the interface image sequence frames and abnormal distribution information.

10. An automated testing system for industrial UI based on machine vision, characterized in that, include: A three-axis gimbal is mounted on which a stylus and an image acquisition device are fixed. The three-axis gimbal is used to respond to control signals to drive the stylus to perform simulated user operations on the display device under test, and to move the image acquisition device with the stylus to acquire a sequence of interface images covering the changes in the interface before and after the simulated user operations; wherein, the simulated user operations include click actions and swipe actions. The first controller is an MCU, which is configured to generate operation commands and send sampling trigger signals through a wireless communication module. The second controller, using an ARM architecture, is connected to the first controller via a wireless communication module and is configured as follows: Receive and respond to the sampling trigger signal, acquire the interface image sequence acquired by the image acquisition device, and preprocess the interface image sequence. The image analysis model is used to identify the interface state of the preprocessed interface image sequence. The image analysis model integrates image feature extraction and text recognition, and outputs the classification result of normal or abnormal interface. The first controller is further configured to: determine whether to continue executing the next simulated user operation or record an exception log based on the interface state recognition result, and generate a test report after the test is completed.

Images