An edge device-based industrial automation software detection method and device

Abstract

This invention discloses an industrial automation software testing method and apparatus based on edge devices, belonging to the field of industrial automation software testing technology. The method, executed by an edge device, includes: acquiring images of the industrial equipment interface and environmental sensor data; dynamically calculating adaptive rank parameters based on the device's own resource status and data complexity; performing local low-rank adaptive fine-tuning on the visual basis model; extracting and fusing features from images and sensors to obtain multimodal fusion features; identifying user interface elements and their coordinates based on these features; receiving test cases and executing automated operations through a USB-HID controller; recording and comparing results to calculate the inconsistency rate, and deciding whether to trigger retraining or upload parameters to a central server for federated learning. This invention achieves non-intrusive, highly robust, adaptive, and offline-supporting automated testing of closed industrial systems, solving the testing challenges under conditions of resource heterogeneity, network instability, and harsh environments.

Description

Technical Field

[0001] This invention relates to the field of industrial automation software testing technology, and specifically to an industrial automation software testing method and apparatus based on edge devices. Background Technology

[0002] Industrial automation software, such as Supervisory Control and Data Acquisition (SCADA) and Human-Machine Interface (HMI) systems, is the core of industrial control, and its stable operation is crucial. However, current testing methods for this type of software have many limitations.

[0003] Automated detection based on frameworks like Selenium relies on access permissions to the internal structure of software interface elements (such as the DOM tree and control handles). Furthermore, many commercial SCADA / HMI systems used in industrial settings are closed systems that do not expose such interfaces, making this method difficult to apply. Even if it could be applied, extensive rewriting and maintenance of detection scripts would be required when the user interface (UI) is updated.

[0004] Computer vision-based automated UI inspection technology can bypass reliance on internal structures by capturing screenshots and using object detection algorithms to identify UI elements. However, in harsh environments such as industrial sites with insufficient lighting and screen glare, the accuracy of purely visual methods drops significantly, and they cannot correlate the causal relationship between device operating status (such as CPU temperature and vibration) and UI anomalies.

[0005] In terms of model optimization, the traditional LoRA (Low-Rank Adaptation) fine-tuning method uses a fixed-rank parameter, which is set based on experience or offline evaluation, typically within the range of 8 to 64. However, the hardware resources of edge devices in industrial settings vary greatly, ranging from high-performance industrial PCs to resource-constrained embedded devices (such as Raspberry Pis). This fixed-rank parameter setting cannot accommodate such differences: running a high-rank model on low-resource devices may lead to memory overflow; while using a low-rank parameter on complex datasets may result in underfitting of the model, causing wasted computational resources or performance degradation.

[0006] In terms of system architecture, existing centralized federated learning or cloud-based training models require a continuous and stable network connection between devices and servers for real-time parameter synchronization. However, network environments in industrial settings such as offshore platforms and mines are often unstable, and frequent network outages can render the system unusable. Furthermore, uploading raw image data containing sensitive information to the cloud also carries risks of high bandwidth costs and data breaches.

[0007] Therefore, achieving efficient and automated testing of industrial automation software without relying on software source code, adapting to diverse hardware resources, coping with unstable network environments, and improving robustness in harsh environments has become a major problem that urgently needs to be solved in current technology. Summary of the Invention

[0008] The purpose of this invention is to provide an industrial automation software inspection method and apparatus based on edge devices, so as to overcome the above-mentioned problems of the prior art. It can adapt to edge devices with different hardware resources without accessing the internal structure of industrial equipment software, support long-term offline operation, and improve the detection accuracy and robustness in harsh industrial environments by fusing multi-source data.

[0009] To achieve the above objectives, the technical solution adopted by the present invention is as follows: An industrial automation software inspection method based on edge devices is applied to a system comprising a central server and at least one edge device, wherein the edge device performs the following steps: S1: Collect image data from the display interface of the industrial equipment and simultaneously collect sensor data from the operating environment of the industrial equipment; S2: Dynamically calculate the adaptive rank parameter based on the current available memory, CPU load, network bandwidth utilization of the edge device, and the dataset complexity calculated based on the image data and sensor data; S3: Obtain the pre-trained visual base model from the central server, and perform low-rank adaptive fine-tuning on the visual base model locally based on the adaptive rank parameter to obtain the fine-tuned local model. During fine-tuning, the original weight matrix of the visual base model is frozen, and only the newly added low-rank decomposition matrix is ​​trained. S4: Use the fine-tuned local model to extract the visual features of the image data and extract the features of the sensor data, and fuse the visual features and the features of the sensor data to obtain multimodal fusion features; S5: Based on the multimodal fusion features, identify the user interface elements and their coordinates in the display interface; S6: Receive test cases from the central server, and execute corresponding automated operations through the USB-HID controller based on the test cases and the identified user interface element coordinates; S7: Record the interface image after the automated operation is executed as the actual result, and compare it with the expected result in the test case to calculate the inconsistency rate; S8: If the inconsistency rate exceeds a preset threshold, the dynamic retraining process is triggered, and the process returns to step S2; otherwise, the test results and the parameters of the low-rank decomposition matrix are uploaded to the central server.

[0010] Furthermore, the calculation of the adaptive rank parameter in S2 satisfies the following: its value is positively correlated with the available memory and the complexity of the dataset, negatively correlated with the CPU load and the network bandwidth utilization, and does not exceed a preset rank upper limit.

[0011] Furthermore, the low-rank adaptive fine-tuning described in S3 satisfies: Where W is the original weight matrix of the visual basis model, B and C are the newly added low-rank decomposition matrices, r is the adaptive rank parameter, x is the input, and h is the output. This is the scaling factor.

[0012] Furthermore, the fusion described in S4 specifically includes: The local model is a visual Transformer model, and the visual features are extracted through the visual Transformer model; The features of the sensor data are obtained by mapping through a fully connected layer; The visual features are combined with the features from the sensor data to obtain the multimodal fusion features.

[0013] Furthermore, the identification of user interface elements and their coordinates described in S5 employs a multi-method fusion strategy, including: For the multimodal fusion features, object detection based on convolutional neural networks, icon recognition based on template matching, and text recognition based on optical character recognition are performed in parallel to obtain the corresponding candidate detection results; The candidate detection results are filtered by applying the corresponding confidence thresholds respectively; For the candidate detection results that overlap after filtering, a non-maximum suppression algorithm is applied to remove duplicates and fuse them to obtain the user interface elements and their coordinates.

[0014] Furthermore, the automated operations described in S6 specifically include: The USB-HID controller is a Teensy development board, and the edge device sends control commands to the Teensy development board via serial communication. The Teensy development board converts the instructions into standard USB-HID messages and sends them to the industrial equipment.

[0015] Furthermore, the method further includes the following steps: S9: When there are multiple edge devices, the central server receives the parameters of the low-rank decomposition matrix uploaded by each edge device, calculates the weighting coefficients based on the amount of data used for local training of each edge device, aggregates the parameters through a federated averaging algorithm, updates the parameters of the visual base model to form an updated global model, and distributes the updated global model to each edge device.

[0016] Furthermore, the sensor data includes temperature, humidity, and vibration acceleration; when any of the sensor data exceeds a preset anomaly threshold, the associated image data is marked as a high-priority sample for the dynamic retraining process.

[0017] Furthermore, the step of concatenating the visual features with the features of the sensor data specifically includes: during or after concatenating the visual features with the features of the sensor data, applying an attention mechanism to dynamically allocate feature weights; when the sensor data indicates that the industrial equipment is in an abnormal environment, the attention mechanism adaptively increases the weight of the sensor data features in the multimodal fusion features.

[0018] Another object of the present invention is to provide an industrial automation software testing device based on edge devices, the device being used to implement the aforementioned industrial automation software testing method based on edge devices, comprising a central server and at least one edge device, wherein: The central server is used to generate test cases and aggregate model parameters from edge devices to update and distribute the global model. The edge device is used to perform automated inspection of industrial equipment and includes: The data acquisition module is used to acquire interface images and environmental sensor data of the industrial equipment; The adaptive model fine-tuning module is used to dynamically calculate the adaptive rank parameter and perform local fine-tuning on the visual base model obtained from the central server based on the edge device's own state and the collected data. The feature fusion and inference module is used to extract and fuse features from the image and sensor data using the fine-tuned model, and to identify user interface elements and their coordinates based on the fused features. The control execution module is used to perform automated operations on the industrial equipment through a USB-HID controller based on the test cases and the identified user interface elements and their coordinates. The decision and communication module is used to calculate the inconsistency rate based on the operation results to decide whether to trigger local model retraining, and to communicate data and parameters with the central server.

[0019] Compared with the prior art, the present invention has the following beneficial effects: First, by dynamically calculating adaptive rank parameters, the model complexity can be automatically adjusted based on the real-time memory, CPU load, and data characteristics of the edge device. On resource-constrained devices such as Raspberry Pi, the rank can be automatically reduced, for example, from a higher fixed rank to a lower adaptive rank, reducing model memory usage by approximately 40%. On high-performance devices, the rank can be adaptively increased, improving single-inference speed by 2.5 times. The system achieves a high degree of automated detection and resource adaptation, resulting in a significant improvement in efficiency.

[0020] Second, by employing a multimodal data fusion and multi-method fusion detection strategy involving images and sensors, the recognition performance is significantly improved. Even in adverse environments such as low light or glare, the accuracy or recall rate of UI element recognition can reach over 98%, an improvement of over 12% compared to pure vision methods, while the false detection rate is controlled below 2%, demonstrating high accuracy and robustness.

[0021] Third, it supports offline operation of edge devices for more than 72 hours, and automatically completes data synchronization after the network is restored. It adopts a federated learning architecture that only uploads model parameters, reducing communication traffic by more than 85%, and uses encrypted channels to ensure data security, perfectly adapting to the unstable networks in industrial sites.

[0022] Fourth, it achieves non-intrusive, driverless control via the USB-HID protocol, compatible with mainstream industrial operating systems. It supports various hardware solutions from Raspberry Pi to high-performance industrial PCs, and based on an open-source software stack, significantly reducing deployment and maintenance costs. Suitable for various industrial automation scenarios, it offers flexible deployment and strong versatility. Attached Figure Description

[0023] Figure 1 This is a flowchart of the method of the present invention. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0025] This invention provides an industrial automation software testing method and apparatus based on edge devices, aiming to solve three core challenges faced in the automated testing of closed SCADA / HMI systems in industrial settings: the inability to intrude into the software to obtain control information; heterogeneous hardware resources and unstable network environments of edge devices; and harsh physical environments, such as insufficient light and vibration, leading to a decrease in the reliability of purely visual recognition. This solution integrates adaptive edge computing, multimodal perception, and federated learning to achieve a non-intrusive, highly robust, and offline-capable automated testing method and apparatus.

[0026] This embodiment provides an industrial automation software inspection method based on edge devices, applied to a system containing a central server and at least one edge device, wherein the edge device performs the following steps: S1: Collect image data from the display interface of the industrial equipment and simultaneously collect sensor data from the operating environment of the industrial equipment; S2: Dynamically calculate the adaptive rank parameter based on the current available memory, CPU load, network bandwidth utilization of the edge device, and the dataset complexity calculated based on the image data and sensor data; S3: Obtain the pre-trained visual base model from the central server, and perform low-rank adaptive fine-tuning on the visual base model locally based on the adaptive rank parameter to obtain the fine-tuned local model. During fine-tuning, the original weight matrix of the visual base model is frozen, and only the newly added low-rank decomposition matrix is ​​trained. S4: Use the fine-tuned local model to extract the visual features of the image data and extract the features of the sensor data, and fuse the visual features and the features of the sensor data to obtain multimodal fusion features; S5: Based on the multimodal fusion features, identify the user interface elements and their coordinates in the display interface; S6: Receive test cases from the central server, and execute corresponding automated operations through the USB-HID controller based on the test cases and the identified user interface element coordinates; S7: Record the interface image after the automated operation is executed as the actual result, and compare it with the expected result in the test case to calculate the inconsistency rate; S8: If the inconsistency rate exceeds a preset threshold, the dynamic retraining process is triggered, and the process returns to step S2; otherwise, the test results and the parameters of the low-rank decomposition matrix are uploaded to the central server.

[0027] The above method and process will be explained in detail below.

[0028] This invention employs a distributed architecture that integrates cloud, edge, and terminal. Logically, the system consists of three layers: a central server at the backend, edge devices deployed at various industrial sites, and the industrial equipment (terminals) acting as the monitoring targets. Edge devices connect to industrial equipment non-intrusively via physical signals and logically connect to the central server via the enterprise intranet or the internet, supporting breakpoint resumption. The central server, acting as the system's brain, can be deployed in a local data center or a private cloud. Its core responsibilities include storing and versioning large pre-trained AI models (base models); automatically parsing equipment manuals and generating test cases based on natural language processing technology; and continuously optimizing the global model by aggregating the experiential knowledge of each edge device while protecting data privacy. Specifically, the test case generation module of the central server is implemented through the following automated process: obtaining the electronic user manual of the industrial equipment; extracting all text content from the manual using optical character recognition (OCR) technology (such as PaddleOCR); and inputting the extracted text into a pre-trained large language model, such as a Transformer-based model, for deep analysis. This Natural Language Processing (NLP) model can understand operation flow descriptions, interface element functions, and state transition logic, and automatically output a structured test case sequence. Each test case includes at least clearly defined operation steps, operation objects (such as UI elements), and the expected result after each step. The system's typical hardware configuration can utilize servers equipped with high-performance GPUs, such as NVIDIA A100, to meet the needs of model training and big data processing. Edge devices, acting as the system's senses and execution arms, are key units deployed at the front lines. They need to possess sufficient computing power, rich I / O interfaces, and good adaptability to industrial environments. This invention provides three flexible hardware selection options, allowing users to choose based on cost, computing power, and environmental requirements: (1) High-performance solution: The system adopts embedded AI platforms such as NVIDIA Jetson AGX Orin, which provide powerful local inference and training computing power. For example, the INT8 computing power can reach 275 TOPS, which is suitable for scenarios with extremely high requirements for real-time detection.

[0029] (2) Economical solution: The system uses Raspberry Pi 4B and other development boards, which are low cost and consume only 3-8W of power. It is suitable for large-scale deployment or cost-sensitive scenarios.

[0030] (3) Industrial-grade solution: The system adopts an industrial control computer with a wide temperature range and high protection level (such as IP65), which can operate stably in extreme temperatures and harsh environments from -40℃ to 85℃.

[0031] Regardless of the hardware used, edge devices must integrate an image acquisition interface, a sensor bus interface, a USB master / slave interface, and a network interface.

[0032] Specifically, let's take connecting a Jetson AGX Orin to an industrial PC (industrial equipment) as an example. First, the video output of the industrial PC is split in two using an HDMI screen splitter. One path connects to the original monitor, and the other path is converted to a USB signal via an HDMI capture card and connected to the Jetson. The Teensy 4.0 development board is connected to the Jetson's USB port via a Micro-USB cable, and its Type-A USB port is connected to the industrial PC's USB port. The DHT22 temperature and humidity sensor and the ADXL345 vibration sensor are connected to the Jetson's GPIO pins via I2C and SPI buses, respectively. This connection method requires no software or driver installation on the industrial PC, achieving completely non-intrusive integration.

[0033] The following combination Figure 1 Each step of the detection method performed by the edge device is described in detail. This method forms a complete process from perception, decision-making, execution to self-optimization.

[0034] Step 1: Collect image data from the display interface of the industrial equipment, and simultaneously collect sensor data from the operating environment of the industrial equipment.

[0035] The goal of this step is to acquire multimodal raw data for sensing the status of industrial equipment, simultaneously ensuring the spatiotemporal alignment of image information with the physical environment status in subsequent analysis. Image data acquisition is performed through the image acquisition module of an edge device, which is responsible for visually reading the human-machine interface of the industrial equipment. Three implementation paths are provided, including: Path 1: Direct HDMI signal acquisition for optimal image quality. This solution utilizes a screen splitter to losslessly copy the HDMI digital signal. For example, using the J-Tech Digital JTD18G-EDID screen splitter ensures that industrial equipment outputs its native optimal resolution, such as 1920x1080, without display anomalies caused by the acquisition process. The split signal is converted into a USB video stream (UVC protocol) via a capture card such as the Elgato Cam Link 4K. Edge devices, through the cv2.VideoCapture interface of the OpenCV library, can stably acquire an uncompressed, latency-free RGB image array at a frame rate of 30fps. This array, such as... Here, 1080 represents the image height (number of pixel rows), 1920 represents the image width (number of pixel columns), and 3 represents the number of image channels (3 for color). This method produces the highest image quality and is the preferred choice for recognizing complex user interface (UI) elements. Option two: Use a USB camera for shooting, offering flexible deployment. When an HDMI signal is unavailable, a USB camera such as the Logitech C920 can be used to shoot directly at the screen. Note that lighting and angle need to be adjusted to reduce reflections. While this method introduces image distortion, chromatic aberration, and moiré patterns, it is extremely simple to deploy and has the lowest cost. Path 3: Software screen capture, suitable for devices that support remote desktop. If the industrial equipment is running Windows / Linux and has VNC or RDP services enabled, software screen capture can be performed using the corresponding client library. This method does not rely on additional hardware, but may be affected by network and system load, resulting in generally lower frame rates and stability.

[0036] The adaptability design of the above-mentioned multiple acquisition methods ensures that the present invention can flexibly cope with the equipment interface limitations and deployment conditions of various industrial sites, and realizes the universal applicability of the method.

[0037] Next, sensor data acquisition is performed. The sensor interface module acquires environmental data via a standard industrial bus protocol, providing crucial physical context for visual perception. In this embodiment, the DHT22 sensor is periodically read via an I2C bus with address 0x27 and a clock speed of 100kHz to obtain the values ​​of temperature T and humidity H, with the periodicity being approximately 10 times per second. The ADXL345 triaxial accelerometer is read via a 5MHz SPI bus to obtain instantaneous acceleration values ​​in the x, y, and z directions. , , Raw sensor readings may contain noise. Moving average filtering can be applied to temperature and humidity data, and Gaussian filtering can be applied to smooth vibration data, with a standard deviation σ=1.5. After preprocessing, sensor data collected at the same time are packaged into a single feature vector, such as... and images with the same timestamp Perform association and binding. Apply predefined rules in real time for anomaly detection, such as if the temperature T > 75℃ or the acceleration amplitude in any direction is [not specified]. >15g, where sqrt() is used to calculate the square root, immediately associating that time point. Data pairs are labeled as high-priority anomalous samples. This mechanism directly correlates anomalous events in the physical world with the performance of the software UI, providing valuable data labels for the model's focused learning and root cause analysis later in the text. These anomalous events include equipment overheating and severe vibration.

[0038] Step 2: Dynamically calculate the adaptive rank parameter based on the current available memory, CPU load, network bandwidth utilization of the edge device, and the dataset complexity calculated based on the image data and sensor data.

[0039] This step is the decision-making stage of the adaptive rank LoRA fine-tuning method. Its purpose is to intelligently match the complexity (number of parameters) of model fine-tuning with the current real-time load of the edge device and the inherent difficulty of the data being processed, thereby fundamentally solving the problem of resource waste or insufficient performance caused by the one-size-fits-all approach of the fixed rank method.

[0040] In this embodiment, before each planned model fine-tuning, the edge device dynamically collects three key resource metrics: currently available memory, CPU load, and network bandwidth utilization. The currently available memory is obtained through system calls, representing the amount of currently idle physical memory in MB. The CPU load is the average CPU utilization over the past second, expressed as a percentage. The network bandwidth utilization assesses the current network upload channel's busy level, also expressed as a percentage. High utilization may indicate heavy synchronization tasks, and the complexity of this training session should be appropriately reduced to quickly release resources.

[0041] For the batch of learning data collected in step 1, such as the most recent 100 image-sensor data pairs, a comprehensive complexity index is calculated. Specifically, this is achieved by calculating the weighted sum of multiple low-level features (such as edge density, color distribution entropy, and density of detected potential UI regions) for each image sample, resulting in a complexity score for that sample. Then calculate all of this batch of samples variance of values The larger the variance, the greater the inherent variability and the higher the learning difficulty of this batch of data.

[0042] Substitute the four inputs above into a predefined decision function. The rank of this function should be positively correlated with available memory and data complexity, negatively correlated with CPU load and network bandwidth utilization, and should have an upper limit. Its calculation formula is as follows: r in, The rank parameter is a preset upper limit value, which is the value set in this embodiment. =64; The current available memory for the edge device; This is a quantitative indicator of the complexity of the currently collected dataset, specifically the standard deviation of the data features; The current CPU load of the edge device; This represents the current network bandwidth utilization of the edge device. These are the empirical weight coefficients of the four factors mentioned above, and and A negative value for the rank parameter r results in a negative correlation between the rank parameter r and CPU load and network bandwidth utilization. In this embodiment, =0.4, =50, =-0.3, =-0.2. The above formula dynamically and comprehensively considers the real-time resource status of edge devices ( , , ) and the characteristics of the data being processed ( An optimal rank parameter is calculated through weighted calculation. This allows for the optimal match between model complexity and device performance under resource constraints.

[0043] This adaptive mechanism produces excellent resource optimization results; for example, on a Raspberry Pi with only 4GB of available memory, the formula may be automatically output. =8, which significantly reduces the number of parameters added in subsequent LoRA fine-tuning, successfully reducing the model's runtime memory usage from approximately 8GB under fixed high rank to 4.8GB, a 40% reduction, perfectly avoiding memory overflow crashes. The fixed high rank is such as r=64. Conversely, on a Jetson device with ample memory and low load, the formula might output r=32, allowing the model to use more parameters to capture data details, thereby increasing single inference speed by 2.5 times and achieving on-demand allocation of computing resources.

[0044] Step 3: Obtain the pre-trained visual basis model from the central server, and based on the adaptive rank parameter, perform low-rank adaptive fine-tuning on the visual basis model locally to obtain the fine-tuned local model. During fine-tuning, the original weight matrix of the visual basis model is frozen, and only the newly added low-rank decomposition matrix is ​​trained. This step is the specific execution of adaptive decision-making, achieving efficient model personalization at resource-constrained edge environments.

[0045] First, the model is acquired and loaded. The edge device downloads the latest visual base model, such as VisionTransformer Base / 16, and its current global LoRA parameters from the central server. This is a powerful model pre-trained on massive amounts of general-purpose images, possessing a good foundation for feature extraction. Then, adaptive LoRA fine-tuning is performed. Unlike traditional LoRA, this is based on the parameters calculated earlier. The fine-tuning structure is dynamically constructed. The original weight matrix W of the base model is loaded and frozen so that it is not updated during training. Then, a pair of low-rank matrices is injected into the bypass layer of the base model. and The key layers are, for example, the Query and Value projection layers of the Transformer. Matrix B is initialized with random Gaussian matrix, and C is initialized as a zero matrix to ensure that the model output is consistent with the original base model at the start of training.

[0046] For efficient training, the computation of this layer during forward propagation becomes: Where W is the original weight matrix of the visual basis model; B and C are the newly added low-rank decomposition matrices; r is the adaptive rank parameter; x is the input; and h is the output. is the scaling factor, a trainable hyperparameter with an initial value of 0.4, used to control the contribution of low-rank updates to the original weights.

[0047] During backpropagation, only the gradients of B and C are calculated and updated. Because... Typically much smaller than the model dimensions d and k (e.g., 8 and 768), the number of trainable parameters added in this fine-tuning is extremely small, possibly only 1 / 50 of the total fine-tuning.

[0048] Training is performed in a small number of epochs, such as 3 epochs, using data collected locally on the edge device that includes labeled hard and outlier examples. The goal is to minimize the UI element recognition loss, enabling the model to quickly adapt to the specific interface style and environmental interference of the current device. This approach of freezing the backbone and training the side path achieves rapid model adaptation to the current scenario with extremely low computational and storage overhead, allowing edge devices to maintain the ability to continuously optimize the model even under resource-constrained conditions.

[0049] Step 4: Extract visual features from the image data using the fine-tuned local model, and extract features from the sensor data. Fuse the visual features with the sensor data features to obtain multimodal fusion features. This process embodies the invention of the multimodal data fusion recognition method, which constructs a more comprehensive and robust perception of device status by fusing visual and physical sensing information.

[0050] Real-time captured interface images The local model, fine-tuned for the current environment and obtained in step 3, is then fed into the model. In this embodiment, the Vision Transformer (ViT) model is used. The forward propagation of the model outputs a high-dimensional, semantically rich feature vector, for example... This vector encodes the type, layout, and visual context information of all UI elements in the current image.

[0051] Then the synchronously acquired sensor data vector The input is a lightweight, fully connected neural network layer. The network's role is to map the raw, low-dimensional sensor readings to a higher-dimensional, more expressive feature space, such as... The sensor readings include temperature, humidity, vibration, etc. This mapping process can capture the complex interactions between multiple sensor readings.

[0052] Next, the two feature vectors from different modalities are concatenated: Where 896 = 768 + 128, 768 comes from the image features extracted by the Vision Transformer (ViT) model, such as the ViT model output feature dimension of 768; 128 comes from the feature vector obtained after the sensor data is mapped through a fully connected layer.

[0053] As an optimization of the basic splicing method described above, an attention mechanism can be introduced during the feature fusion process to adaptively adjust the contribution weights of different modal features in the final decision. Specifically, after splicing to obtain the joint representation... Then, it can be fed into a lightweight attention network. This network can dynamically calculate visual features based on the current input context. With sensor characteristics The importance weighting of sensor features is emphasized. Especially when sensor data indicates abnormal environmental conditions, such as severe vibration or high temperature, the attention mechanism can automatically increase the weight of sensor features in the fused features. This allows the system to rely more on reliable physical sensing signals for judgment when visual information may be distorted due to environmental interference, thereby further improving the robustness and accuracy of the fusion decision.

[0054] The resulting It is a joint representation that simultaneously contains information about both "what the screen is displaying" and "the physical environment in which the device is located." This fusion strategy is key to improving the system's robustness in harsh environments. For example, when the ambient light suddenly dims, causing image... Features of a pure visual model when quality is severely degraded It may fail. However, at this time, if the vibration sensor characteristics... The display device continuously vibrates at high frequency, and the fusion feature... The sensor portion of the image becomes a strong signal, guiding the system to be alert to potential device anomalies, thus avoiding misjudgments based solely on distorted visual information. Experiments show that this fusion method can improve UI recognition accuracy by more than 12% under harsh operating conditions.

[0055] Step 5: Based on the multimodal fusion features, identify the user interface (UI) elements and their coordinates in the display interface. This step employs a multi-method fusion detection strategy to analyze the fusion features and ultimately accurately output the position and category of the UI elements.

[0056] In this embodiment, the system runs three detection engines in parallel to understand the fused features or the original image from different perspectives, including: (1) Convolutional Neural Network (CNN) object detection engine. Based on frameworks such as YOLO, it directly processes the visual part of the image or fused features, and outputs a series of bounding box suggestions with confidence, in the following format: It excels at detecting general-purpose controls with regular shapes.

[0057] (2) Template matching engine. Loads a pre-stored library of standard button and icon templates for this model of industrial equipment. Calculates similarity scores such as normalized cross-correlation using a sliding window to locate regions in the image that highly match the templates. Highly effective for recognizing manufacturer-specific, stylized icons.

[0058] (3) Optical Character Recognition (OCR) text recognition engine. Using tools such as PaddleOCR, it recognizes all text regions and their content in the image. This is crucial for recognizing text boxes, labels, status prompts, and other information.

[0059] Next, adaptive threshold filtering and decision fusion are performed. First, threshold filtering is applied because not all detection results are reliable. The system sets adaptive thresholds for each method: for example, only results with CNN detection confidence > 0.6, template matching similarity > 0.8, and OCR confidence > 0.7 are retained. After threshold filtering, different methods may produce multiple overlapping detection boxes for the same UI element. At this point, a non-maximum suppression algorithm is applied, filtering based on overlap (e.g., IoU > 0.5) and confidence / similarity, retaining only the optimal detection result. The final results from the three methods, after filtering and deduplication, are merged into a unified list of UI elements. Each element contains its precise center coordinates (…). The strategy utilizes the bounding box, category, and source of recognition, such as "button," "text box," and "warning icon." This approach leverages the strengths of different recognition methods, creating a complementary system. CNN ensures high recall, template matching improves accuracy for specific icons, and OCR ensures no text information is missed. Ultimately, this results in an overall recall rate of over 98% for UI elements, while keeping the false positive rate below 2%, achieving the extremely high reliability required for industrial applications.

[0060] Step 6: Receive test cases from the central server, and execute corresponding automated operations through the USB-HID controller based on the test cases and the identified user interface element coordinates. This process is the execution phase, transforming the virtual identification results into real device operations; its core is the USB-HID driverless control method.

[0061] Specifically, edge devices receive structured test cases from the central server, such as "1. Click the 'Login' button; 2. Enter 'admin' in the 'Username' box;...". These test cases are automatically generated from the device manual by the server-side Natural Language Processing (NLP) module, without the need for manual scripting.

[0062] Perform coordinate mapping and command generation. The edge device parses use cases, such as "click the 'Login' button". It searches the UI element list generated in step 5 for a button element with the text "Login" and obtains its center coordinates (…). This then generates a series of low-level control instructions, such as MOUSE_MOVE( );DELAY(100);MOUSE_CLICK(LEFT).

[0063] Next, USB-HID protocol conversion and injection are performed. The edge device sends the above instructions to the Teensy 4.0 development board connected to it via a serial port (UART, 115200 baud rate). The Teensy development board has custom firmware burned on it; its core function is to translate the received serial port instructions into standard USB-HID messages natively supported by the operating system in real time. Teensy, through its USB Type-A interface, simulates itself as a standard USB keyboard and mouse, directly plugging into the USB port of the industrial device. The operating system of the industrial device, such as Windows / Linux / VxWorks, will recognize it as a regular peripheral and execute the mouse movement, click, or keyboard input operations corresponding to the received HID messages. This method achieves single-pixel-level operational precision with an operation latency of less than 50 milliseconds. Since HID is an operating system-level standard protocol, the biggest advantage of this process is that it requires no driver. Therefore, no software, drivers, or agents need to be installed on the industrial device under test, achieving true zero-intrusion, configuration-free automation, greatly improving the convenience and versatility of deployment.

[0064] Step 7: Record the interface image after the automated operation is executed as the actual result, and compare it with the expected result in the test case to calculate the inconsistency rate. This process is a verification step, used to evaluate whether a single test operation is successful and to provide quantitative indicators for system self-optimization.

[0065] After performing the automated operation in step 6, the system waits for a short reaction time, such as 500ms, to allow the industrial equipment interface to stabilize. Immediately afterwards, an image acquisition is triggered to capture the actual screen image after the operation is executed. Actual footage Compare the result with the expected result in this test case. The expected result can take several forms: (1) A descriptive text, such as "a success dialog box should pop up", can be used to judge the text in the actual screen by combining OCR recognition; (2) A standard screenshot, at which point image comparison algorithms such as structural similarity index can be used to calculate similarity.

[0066] If the comparison results meet the success criteria, such as text matching and similarity exceeding the threshold, then the step is considered successful; otherwise, it is considered a failure.

[0067] Next, the inconsistency rate is calculated, defined as the proportion of steps where the actual result does not match the expected result out of the total number of steps. For example, after completing a set of test cases containing 20 steps, the number of all failed steps is counted. The formula for calculating the inconsistency rate is: in, The expected result refers to the pre-defined state of the industrial equipment interface that should appear after the successful execution of the t-th test step, which comes from the structured test cases automatically generated by the central server. The actual result refers to the state of the industrial equipment human-machine interface (HMI / SCADA) that the system actually captures after the t-th test step is executed; This represents the total number of steps contained in a test case. This is an indicator function that returns 1 if the condition is true and 0 otherwise. It checks each test step t: if the actual result... Expected results Inconsistencies, such as expecting a "success" message but actually receiving an "error" alert, are counted as 1, representing a failed step; if they are consistent, they are counted as 0. This ratio quantifies the current system's performance in this test and is a key indicator for triggering subsequent optimization decisions.

[0068] Step 8: If the inconsistency rate exceeds a preset threshold, trigger the dynamic retraining process and return to step S2; otherwise, upload the test results and the parameters of the low-rank decomposition matrix to the central server. This process is a decision-making and evolutionary stage, comprising a dynamic retraining triggering mechanism and a distributed federated learning architecture, enabling the system to have self-repair and collective learning capabilities.

[0069] The dynamic retraining decision specifically involves the system pre-setting an inconsistency rate alarm threshold, for example, 0.05. If the inconsistency rate calculated in step 7... Exceeding this threshold indicates that the current local model may be experiencing performance degradation due to interface changes, environment drift, or encountering new problems, and the system will trigger a dynamic retraining process. This is the core condition for triggering retraining.

[0070] To further optimize and prevent frequent and unnecessary retraining caused by occasional misidentification or brief interface flickering, this invention also provides a more robust composite triggering strategy. In this preferred strategy, in addition to satisfying the core condition that the inconsistency rate exceeds a preset threshold, the system can further determine whether the following two additional conditions are simultaneously met: (1) Core condition: The calculated inconsistency rate exceeds the preset threshold (e.g., 0.05); (2) The time interval between the last successful execution of the dynamic retraining process is greater than the preset anti-shake time threshold (e.g., 1 hour). (3) New samples that do not match the expected results are difficult cases, and the number of such samples reaches the preset lower limit (e.g., 10).

[0071] The system only performs dynamic retraining immediately when both the core condition (1) and the additional conditions (2) and (3) are met. If only the core condition (1) is met, but the additional conditions (2) or (3) are not met, the system will cache the currently identified problematic sample data locally and wait for subsequent accumulation or time conditions to be met before triggering the system uniformly. This composite triggering strategy is a preferred embodiment of the present invention, which can effectively avoid overfitting and waste of computational resources while ensuring that the model adapts to real changes in a timely manner.

[0072] The system automatically collects recently identified samples that failed to be identified or had low confidence (i.e., difficult examples) and uses them as new training datasets. The process then jumps back to step S2, recalculates the adaptive rank parameters with the latest device status and this new batch of data, and starts a new round of LoRA incremental fine-tuning, which usually only takes 3 epochs.

[0073] After retraining, the effect is immediately verified with a small number of tests. If the inconsistency rate improves, the local model is updated; if it is ineffective or even deteriorates, it is automatically rolled back to the previous model version. This mechanism gives the system long-term self-maintenance capabilities, enabling it to automatically adapt to changes and keep the inconsistency rate of recognition performance stable at a low level, such as below 0.02, in the long term, significantly reducing the need for manual intervention by maintenance personnel.

[0074] In addition, the method of the present invention further includes the following steps: Step 9: When there are multiple edge devices, the central server receives the parameters of the low-rank decomposition matrix uploaded from each edge device, calculates the weighting coefficients based on the amount of data used for local training of each edge device, aggregates the parameters through a federated averaging algorithm, updates the parameters of the visual base model to form an updated global model, and distributes the updated global model to each edge device.

[0075] First, parameters are uploaded. If the local test performance is normal (i.e., the inconsistency rate is below the threshold), the edge device will not upload any raw images or sensor data. Instead, it will package the test logs, results, and the low-rank factorization matrix parameters (B, C) after training or updating. These parameter files are very small, only about 200KB per update. Uploading only model parameters strictly protects the privacy of raw data in the industrial field, and compared to uploading raw images (each potentially 2MB), communication traffic is reduced by more than 85%. Data is transmitted securely via encrypted channels such as MQTT over TLS.

[0076] The central server's federated learning parameter aggregation module periodically collects parameter updates from multiple edge devices, such as every 24 hours. A federated averaging algorithm is used to assign weights to the parameters of each device based on the amount of local training data, performing a weighted average to integrate the experience of all devices and update the LoRA parameters of the global visual base model, forming a more powerful global model. The aggregation process using the federated averaging algorithm includes: receiving the local LoRA fine-tuning parameters (i.e., low-rank decomposition matrices) uploaded by all online edge devices k in the current communication round. and and the number of data samples used for local training on each device. ; Calculate the total data volume Then calculate the aggregate weight for each device. This means that devices with larger amounts of data contribute more to the global model; based on weights... The LoRA parameters of each device are weighted and averaged to update the global parameters. The aggregation formula is as follows: use and Update the global LoRA parameters of the central server to form a more powerful global model that incorporates the experience of all edge devices.

[0077] The updated global model is redistributed to all edge devices, forming a complete positive cycle of distributed learning, centralized aggregation, and shared evolution. Each edge device benefits from collective intelligence, improving the generalization ability of its own model, and can also contribute to the global model without sharing local data. This system perfectly adapts to the unstable and frequently offline characteristics of industrial networks, supporting devices to work continuously for more than 72 hours without network access and automatically synchronizing learning results after the network is restored.

[0078] The above description of the industrial automation software detection method based on edge devices of the present invention is based on preferred embodiments. Various substitutions, combinations, or extensions can be made to each technical aspect without departing from the core concept of the present invention.

[0079] Specifically, in the image data acquisition stage, in addition to the aforementioned combination of an HDMI screen splitter and an HDMI capture card, other methods can also be used. For example, a USB camera can be used to directly capture images of the industrial equipment's display interface. For industrial equipment that supports remote desktop protocols, software screen capture can also be performed via VNC or RDP protocols to obtain interface images. These methods provide flexibility to adapt to different device interfaces and on-site deployment conditions.

[0080] In the multimodal data fusion stage, the sensor data is not limited to temperature, humidity, and vibration acceleration. Depending on specific monitoring needs, other types of industrial sensors can be integrated, such as current sensors for monitoring equipment power consumption or sound sensors for analyzing equipment operating status. The multimodal fusion method is also applicable to fusing image data with these extended sensor data.

[0081] When fine-tuning the model on edge devices, the low-rank adaptive (LoRA) fine-tuning method can use a fixed rank parameter. Furthermore, it can be combined with other model lightweighting techniques, such as INT8 quantization to compress the model size and improve inference speed, or knowledge distillation to train a lighter student model. The adaptive rank calculation method can be used in conjunction with these strategies to further optimize resource utilization.

[0082] Communication between edge devices and the central server, as well as between modules within the edge devices, can use the MQTT protocol, or other applicable communication protocols such as HTTP / HTTPS and CoAP, to adapt to different network environments and security requirements.

[0083] This embodiment provides an industrial automation software testing device based on edge devices. The device is used to implement the aforementioned industrial automation software testing method based on edge devices, and includes a central server and at least one edge device, wherein: The central server is used to generate test cases and aggregate model parameters from edge devices to update and distribute the global model. The edge device is used to perform automated inspection of industrial equipment and includes: The data acquisition module is used to acquire interface images and environmental sensor data of the industrial equipment; The adaptive model fine-tuning module is used to dynamically calculate the adaptive rank parameter and perform local fine-tuning on the visual base model obtained from the central server based on the edge device's own state and the collected data. The feature fusion and inference module is used to extract and fuse features from the image and sensor data using the fine-tuned model, and to identify user interface elements and their coordinates based on the fused features. The control execution module is used to perform automated operations on the industrial equipment through a USB-HID controller based on the test cases and the identified user interface elements and their coordinates. The decision and communication module is used to calculate the inconsistency rate based on the operation results to decide whether to trigger local model retraining, and to communicate data and parameters with the central server.

[0084] The data acquisition module acquires the display interface image of the industrial equipment through any of the following methods: a combination of an HDMI screen splitter and an HDMI capture card, a USB camera, or a remote desktop protocol.

[0085] In summary, this invention does not rely on the source code or internal interfaces of industrial automation software (such as SCADA and HMI systems). Through non-intrusive image acquisition and USB-HID control, it is applicable to various closed commercial systems, solving the problem of traditional automated testing methods being unable to be deployed in industrial settings. Through adaptive rank LoRA fine-tuning, multimodal data fusion, and multi-method fusion detection strategies, the system can achieve a detection efficiency of over 300 test cases per hour on high-performance edge devices. Simultaneously, the adaptive mechanism can dynamically adjust model complexity according to device resources. For example, on resource-constrained devices such as Raspberry Pi, it can reduce model memory usage by approximately 40%, effectively avoiding memory overflow and achieving an optimal balance between detection efficiency and resource consumption. By fusing visual and sensor data, the system's UI recognition accuracy in harsh industrial environments such as low light and glare can be improved by more than 12% compared to pure vision methods. The system design supports long-term offline operation of edge devices and automatic synchronization after network recovery, perfectly adapting to the unstable and frequently out-of-network environment of industrial sites. The system possesses dynamic retraining and federated learning capabilities, automatically triggering model optimization based on the detection inconsistency rate and aggregating experiential knowledge into the global model while protecting privacy, achieving continuous and collaborative evolution of system performance. This significantly reduces the manual intervention costs required for long-term system operation and maintenance.

[0086] Embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0087] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0088] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0089] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0090] Contents not described in detail in this specification are prior art known to those skilled in the art. It is hereby indicated that the above description is intended to help those skilled in the art understand this invention, but does not limit the scope of protection of this invention. Any equivalent substitutions, modifications, improvements, or simplifications of the above descriptions that do not depart from the essential content of this invention fall within the scope of protection of this invention.

Claims

1. An industrial automation software inspection method based on edge devices, applied to a system comprising a central server and at least one edge device, characterized in that, The edge device performs the following steps: S1: Collect image data from the display interface of the industrial equipment and simultaneously collect sensor data from the operating environment of the industrial equipment; S2: Dynamically calculate the adaptive rank parameter based on the current available memory, CPU load, network bandwidth utilization of the edge device, and the dataset complexity calculated based on the image data and sensor data; S3: Obtain the pre-trained visual base model from the central server, and perform low-rank adaptive fine-tuning on the visual base model locally based on the adaptive rank parameter to obtain the fine-tuned local model. During fine-tuning, the original weight matrix of the visual base model is frozen, and only the newly added low-rank decomposition matrix is ​​trained. S4: Use the fine-tuned local model to extract the visual features of the image data and extract the features of the sensor data, and fuse the visual features and the features of the sensor data to obtain multimodal fusion features; S5: Based on the multimodal fusion features, identify the user interface elements and their coordinates in the display interface; S6: Receive test cases from the central server, and execute corresponding automated operations through the USB-HID controller based on the test cases and the identified user interface element coordinates; S7: Record the interface image after the automated operation is executed as the actual result, and compare it with the expected result in the test case to calculate the inconsistency rate; S8: If the inconsistency rate exceeds a preset threshold, the dynamic retraining process is triggered, and the process returns to step S2; otherwise, the test results and the parameters of the low-rank decomposition matrix are uploaded to the central server.

2. The industrial automation software inspection method based on edge devices according to claim 1, characterized in that, The calculation of the adaptive rank parameter in S2 satisfies the following: its value is positively correlated with the available memory and the complexity of the dataset, negatively correlated with the CPU load and the network bandwidth utilization, and does not exceed a preset rank upper limit.

3. The industrial automation software testing method based on edge devices according to claim 1, characterized in that, The low-rank adaptive fine-tuning described in S3 satisfies: Where W is the original weight matrix of the visual basis model, B and C are the newly added low-rank decomposition matrices, r is the adaptive rank parameter, x is the input, and h is the output. This is the scaling factor.

4. The industrial automation software testing method based on edge devices according to claim 1, characterized in that, The fusion described in S4 specifically includes: The local model is a visual Transformer model, and the visual features are extracted through the visual Transformer model; The features of the sensor data are obtained by mapping through a fully connected layer; The visual features are concatenated with the features of the sensor data to obtain the multimodal fusion features.

5. The industrial automation software testing method based on edge devices according to claim 1, characterized in that, The identification of user interface elements and their coordinates described in S5 employs a multi-method fusion strategy, including: For the multimodal fusion features, object detection based on convolutional neural networks, icon recognition based on template matching, and text recognition based on optical character recognition are performed in parallel to obtain the corresponding candidate detection results; The candidate detection results are filtered by applying the corresponding confidence thresholds respectively; For candidate detection results that overlap after filtering, a non-maximum suppression algorithm is applied to remove duplicates and fuse them to obtain the user interface elements and their coordinates.

6. The industrial automation software testing method based on edge devices according to claim 1, characterized in that, The automated operations described in S6 specifically include: The USB-HID controller is a Teensy development board, and the edge device sends control commands to the Teensy development board via serial communication. The Teensy development board converts the instructions into standard USB-HID messages and sends them to the industrial equipment.

7. The industrial automation software testing method based on edge devices according to claim 1, characterized in that, The method further includes the following steps: S9: When there are multiple edge devices, the central server receives the parameters of the low-rank decomposition matrix uploaded by each edge device, calculates the weighting coefficients based on the amount of data used for local training of each edge device, aggregates the parameters through a federated averaging algorithm, updates the parameters of the visual base model to form an updated global model, and distributes the updated global model to each edge device.

8. The industrial automation software testing method based on edge devices according to claim 1, characterized in that, The sensor data includes temperature, humidity, and vibration acceleration; when any of the sensor data exceeds a preset anomaly threshold, the associated image data is marked as a high-priority sample for the dynamic retraining process.

9. The industrial automation software inspection method based on edge devices according to claim 4, characterized in that, The step of concatenating the visual features with the features of the sensor data specifically includes: applying an attention mechanism to dynamically allocate feature weights during or after the concatenation process; and adaptively increasing the weight of the sensor data features in the multimodal fusion features through the attention mechanism when the sensor data indicates that the industrial equipment is in an abnormal environment.

10. An industrial automation software inspection device based on edge devices, characterized in that, The apparatus is used to implement an industrial automation software testing method based on edge devices according to any one of claims 1-9, comprising a central server and at least one edge device, wherein: The central server is used to generate test cases and aggregate model parameters from edge devices to update and distribute the global model. The edge device is used to perform automated inspection of industrial equipment and includes: The data acquisition module is used to acquire interface images and environmental sensor data of the industrial equipment; The adaptive model fine-tuning module is used to dynamically calculate the adaptive rank parameter and perform local fine-tuning on the visual base model obtained from the central server based on the edge device's own state and the collected data. The feature fusion and inference module is used to extract and fuse features from the image and sensor data using the fine-tuned model, and to identify user interface elements and their coordinates based on the fused features. The control execution module is used to perform automated operations on the industrial equipment through a USB-HID controller based on the test cases and the identified user interface elements and their coordinates. The decision and communication module is used to calculate the inconsistency rate based on the operation results to decide whether to trigger local model retraining, and to communicate data and parameters with the central server.

Images