Non-intrusive equipment monitoring system design method based on optical character recognition (OCR)

Abstract

The invention relates to the technical field of industrial automation and computer vision, and particularly discloses a non-intrusive equipment monitoring system design method based on optical character recognition (OCR). The method comprises the following steps: firstly, transmitting a video source to an industrial personal computer through a video encoder and a switch; secondly, establishing a virtual keyboard and mouse remote control channel through the development board and the keyboard and mouse simulator; a template matching mechanism is introduced, corresponding characteristic values and content recognition templates are established for different interfaces of the equipment, and then a PP-OCRV4 model is used for carrying out recognition processing on the image; and finally, establishing monitoring to monitor and control the equipment identification process. According to the invention, a modularized system building method is adopted, the function of each module is specifically developed, a monitoring system for automatically identifying the interface data of the storage equipment is reasonably constructed, and the system can ensure long-time high-efficiency operation, accurately identify the data of each interface and store the data.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of industrial automation and computer vision, in particular to a non-invasive device monitoring system design method based on optical character recognition (OCR), which is used to realize automatic collection and recognition of device interface information without modifying the device hardware or communication interface, and is suitable for operation data monitoring and information extraction in various industrial scenarios. BACKGROUND

[0002] Under the background of the wide promotion of Industry 4.0 and digital factory, the operation data of automatic devices as the core information of production process monitoring, quality tracing and device management, its efficient collection and integration capability has become the basic support of industrial informatization system. However, there are still a large number of traditional devices in the current industrial field without standard communication interface or using closed protocol system, which is difficult to access MES, SCADA and other upper systems through conventional means, forming a "data island", which seriously restricts the real-time flow of production information and the improvement of device intelligent operation and maintenance level.

[0003] The existing industrial data collection scheme mainly depends on the following two ways: one is to modify the internal structure of the device, such as adding PLC communication module, collection card, IO expansion interface, etc. to realize the bottom layer data reading; the second is to realize soft access integration through the communication protocol (such as Modbus, OPC UA, Profibus, etc.) opened by the device manufacturer. However, the above methods generally have problems such as complex adaptation, long implementation period, potential interference to device stability, high maintenance threshold, etc., especially in closed systems, black box control logic or customized embedded terminals.

[0004] In order to break through the above limitations, non-invasive image recognition means has gradually become an effective complementary path for industrial data collection. Among them, the method based on optical character recognition (OCR) can realize the structured extraction of screen display data by collecting and recognizing the device interface image, which has the advantages of flexible deployment, zero interference to the device, easy maintenance, etc.

[0005] However, the current most of the industrial recognition schemes based on OCR still face some technical problems: such as the recognition accuracy is greatly affected by the change of interface style, the image collection stability is insufficient, the template adaptation ability is weak, it cannot support interface automatic switching or multi-device cooperation, etc., which leads to its difficulty in meeting the comprehensive requirements of recognition accuracy, automation degree and system stability in industrial scenarios.

[0006] Therefore, it is urgent to build an industrial OCR recognition system with high recognition accuracy, good environmental adaptability, scalability, strong multi-interface recognition capability and support for automatic interaction, to realize real-time, reliable and automatic monitoring of the operation state and key data of the device without affecting the original function of the device. SUMMARY

[0007] In order to solve the problem that the existing OCR-based industrial recognition scheme is difficult to meet the comprehensive requirements of recognition accuracy, automation degree and system stability in industrial scenarios, the application provides a non-intrusive device monitoring system design method based on optical character recognition (OCR). The method collects the video signal output by the device HDMI, pre-processes and regionally calibrates the image, combines the OCR recognition module (feature value template and content recognition template), and cooperates with the dynamic template mechanism and virtual keyboard and mouse interaction strategy, automatically completes multi-interface polling and data structured extraction, realizes non-intrusive data acquisition without affecting the original device hardware and software structure. Finally, the recognition result is uploaded to the database or industrial information system in real time, realizing the data acquisition goal of high stability, high compatibility and high expansion.

[0008] To achieve the above purpose, the application provides a non-intrusive device monitoring system design method based on optical character recognition (OCR), comprising the following steps:

[0009] S1, using a video encoder and a switch to encode and output the device video source, inputting it into a local industrial computer for decoding processing, connecting the industrial computer and the device based on a hardware development board and a keyboard and mouse simulator, realizing remote keyboard and mouse control of the industrial computer to the device, and finally building the overall hardware connection structure;

[0010] S2, establishing a template matching mechanism, setting feature value templates and content recognition templates for different interfaces of the device, establishing a virtual keyboard and mouse operation template, and establishing a picture preprocessing mode template;

[0011] S3, using PaddleOCR framework, selecting PP-OCRV4 model as the main model for OCR processing, mainly using text detection algorithm DB and text recognition algorithm SVTR;

[0012] S4, using a hybrid architecture combining relational database SQL Server and in-memory data storage Redis for data storage;

[0013] S5, based on Pyside6 framework, building the overall industrial computer interface to monitor and control the recognition process.

[0014] Further, the step S1 specifically comprises the following sub-steps:

[0015] S11, hardware construction of video stream transmission:

[0016] The video acquisition part of the system includes an HDMI video encoder, an Ethernet switch, a network cable, and an industrial computer. The HDMI video encoder converts the video signal of the device into an H.264 format video stream, fixes the output resolution at 1920x1080, and pushes it to the network through the RTSP protocol, achieving low-latency, high-compatible video data transmission control;

[0017] The encoded video stream enters the switch via the network cable and is forwarded to the industrial computer according to the MAC address. The wired LAN transmission has the advantages of strong anti-interference ability, high transmission rate, and stable connection, and is suitable for industrial scenarios that require high image continuity and integrity;

[0018] The industrial computer uses the OpenCV tool based on Python to decode the RTSP video stream, generate continuous image frames, and hand them over to the OCR recognition module for processing, achieving key information extraction, analysis, and storage.

[0019] S12, hardware setup of virtual keyboard and mouse:

[0020] This module is composed of a Shineblink hardware development board and a keyboard and mouse emulator CH9329, and establishes a communication link with the industrial computer through a standard USB serial port. The Shineblink hardware development board, as the master control unit, receives software instructions (such as coordinate points, click types, delay times, etc.) from the industrial computer, and drives the keyboard and mouse emulator CH9329 to simulate real mouse or keyboard operations, including single click, double click, sliding, scroll wheel, and combination key actions, etc., after parsing through the serial communication protocol.

[0021] The keyboard and mouse emulator CH9329 supports the USB HID (Human Interface Device) protocol, and can be identified as a physical peripheral on the controlled terminal (such as an industrial computer), without the need to install drivers to complete operation injection. This solution is completely independent of the target device operating system, without the need to install any programs inside the device, and has good system independence and stability, and can achieve efficient, continuous, and automated data acquisition and task scheduling.

[0022] Further, the step S2 specifically includes the following sub-steps:

[0023] S21, establishing a feature value template and a content recognition template:

[0024] The double-layer template matching mechanism is divided into a feature value template and a content recognition template. To recognize character information, the system first calls the feature value template to extract feature values from a preset region in the image, and matches the current features with the established feature template library to identify the current page type and switch to the corresponding content recognition template in the content recognition template library. The content recognition template will recognize the whole picture after cutting according to the coordinate information in the template, greatly accelerating the efficiency of OCR recognition and improving the recognition accuracy.

[0025] To establish the feature template library and the content recognition template library, the system designs an OCR template establishment program interface based on PySide6. The user can freely frame the coordinate region of interest on the picture and name it for saving on this interface. All template files use structured JSON file format for the OCR recognition module (feature value template and content recognition template) to read. The system loads through memory mapping to optimize access speed, supports hot update and condition-triggered dynamic switching during running. To improve robustness, the system can set a confidence interval threshold under fuzzy matching. When the matching confidence is lower than the set value, the standby template or alarm mechanism is triggered.

[0026] S22, establish a virtual keyboard and mouse operation template:

[0027] The virtual keyboard and mouse operation template is designed to control various interface interaction behaviors at the instruction level. The template uses structured JSON format for configuration, defining the following fields: operation type, target coordinates, click delay time, name, etc. The operation type includes clicking and scrolling, and clicking is divided into not clicking, left clicking and right clicking; scrolling is divided into not scrolling, up scrolling and down scrolling. The specific operation form of each type can be automatically matched.

[0028] During running, the template is loaded and interpreted in real time through the instruction scheduler, controls the Shineblink hardware development board to generate corresponding serial port instructions, and drives the keyboard and mouse simulator CH9329 to complete the corresponding physical simulation action.

[0029] Each operation template can be bound to a specific page feature or recognition field to form a logical closed loop.

[0030] S23, establish a picture preprocessing method template:

[0031] To further improve the recognition accuracy and template adaptation effect, a matching picture preprocessing method template mechanism needs to be established. To solve the problem of large image quality fluctuations under different device interfaces, shooting conditions and signal interference, the system allows users to specify a picture preprocessing method template mechanism for each OCR recognition module during template configuration.

[0032] The picture preprocessing mode template supports the following combination of processing operation modules: image grayscale (Grayscale), adaptive thresholding (Adaptive Thresholding), image sharpening (Sharpening), color inversion, edge enhancement (Laplacian / DoG), noise filtering (Gaussian / Median), rotation correction and scale normalization, etc. Each processing operation can configure its parameter value, and the system will execute it step by step according to the order defined in the template before OCR recognition.

[0033] Further, the step S3 specifically comprises the following sub-steps:

[0034] S31, introduction of the text detection algorithm DB of the PP-OCRV4 model under the PaddleOCR framework:

[0035] The text detection algorithm of the PP-OCRV4 model is based on the DB model, but is optimized in multiple dimensions for industrial scenarios, forming the DB++ architecture. The core idea of the DB algorithm is to introduce a differentiable binarization module, making the binarization operation differentiable, so that it can be trained end-to-end.

[0036] S32, network structure of the DB algorithm:

[0037] The PP-OCRv4 model uses PP-LCNetV3 as the backbone network, which provides rich and efficient feature representation for the DB algorithm. The PP-LCNetV3 series model is a continuation of the PP-LCNet series model, covering a larger range of precision and being able to adapt to the needs of different downstream tasks. The PP-LCNetV3 series model is optimized in multiple aspects, proposes a learnable affine transformation module, improves the reparameterization strategy and activation function, and adjusts the network depth and width. Finally, the PP-LCNetV3 series model can achieve the best balance between performance and efficiency, and achieve the best inference speed in different precision ranges.

[0038] S33, differential binarization of the DB algorithm:

[0039] Traditional methods usually need a complex post-processing step to convert the probability map into a binary image when performing text detection. This process is usually not differentiable, so it cannot be optimized with the network. The DB algorithm introduces an approximate binarization function, making this process differentiable. This way, the process can be optimized directly during training, improving the accuracy of text detection.

[0040] Specifically, given a probability map generated by the segmentation network where H and W are the height and width respectively, and DB algorithm uses an adaptive threshold map T to generate an approximate binary map :

[0041] .

[0042] Here, k is a magnification factor used to control the steepness of the function, with a default setting of 50. This method not only helps to distinguish the text region from the background, but also effectively separates closely connected text instances.

[0043] S34, Adaptive Scale Fusion of DB Algorithm:

[0044] Multi-scale feature fusion is crucial for improving the robustness of the model to text instances of different sizes. However, traditional multi-scale fusion methods often directly concatenate or add feature maps of different scales, lacking flexibility. The ASF module is designed to dynamically fuse features from different scales. First, the feature maps of different scales are upsampled to the same resolution and concatenated together; then, the attention weights are calculated through a spatial attention mechanism; finally, these weights are used to weight the feature maps of each scale to obtain the final fused feature map F.

[0045]

[0046] Here, Conv represents a 3x3 convolution operation, concat represents a concatenation operation, and SpatialAttention is a spatial attention mechanism.

[0047] S35, Introduction of SVTR Text Recognition Algorithm of PP-OCRV4 Model under PaddleOCR Framework:

[0048] The recognition algorithm of PP-OCRV4 model is upgraded from the traditional CRNN to SVTR (Scene Text Recognition with Transformers), combining the advantages of CNN and Transformer, significantly improving the recognition ability of long text and complex fonts.

[0049] S36, Pure Transformer Architecture of SVTR Algorithm:

[0050] SVTR abandons the traditional combination of convolutional neural networks and recurrent neural networks, and instead uses Transformer as its core component. This change allows the model to better capture long-range dependencies and does not require a limit on sequence length.

[0051] The core of Transformer is its encoder part, Transformer Encoder is stacked by multiple identical layers, each layer includes a Multi-Head Self-Attention (MHSA) and a Feed-Forward Network (FFN).

[0052] 1) Input: Flatten the feature map into a sequence

[0053] 2) Multi-Head Attention:

[0054]

[0055] Where the calculation of each head is:

[0056]

[0057] 3) Positional Encoding: Use a learnable 2D Positional Encoding to preserve spatial position information.

[0058] S37, Hybrid Attention Mechanism of SVTR Algorithm:

[0059] SVTR introduces a hybrid attention mechanism that combines the advantages of global and local attention. Specifically, in addition to the standard multi-head self-attention, it may also include a local attention mechanism that focuses on specific regions or features, allowing it to better capture detailed information.

[0060] S38, Flexible Input Size Adaptability of SVTR Algorithm:

[0061] Since RNN is not used, SVTR can more flexibly handle text lines of different lengths, which improves the model's adaptability to various text layouts.

[0062] Further, the step S4 specifically comprises the following sub-steps:

[0063] S41, SQL Server is responsible for data persistence storage, ensuring data reliability and integrity:

[0064] The relational database SQL Server in the database system serves as a persistent storage layer, responsible for creating the "OCRdata" database and tables containing fields such as picture_number (primary key), picture_path, context1-context4, created_at, etc. It has the ability to dynamically detect and automatically create tables and fields. It initializes the connection through a connection string and monitors the status. When data is inserted, it is written into the table through SQL statements. If the insertion fails, it will be rolled back and the Redis cache will be deleted to ensure consistency. When querying, Redis is supplemented with missing data. When reading, full data is loaded into Redis to ensure the reliability and integrity of long-term data storage.

[0065] S42, Redis as a cache layer, optimizes data access speed and improves system response performance:

[0066] The relational database SQL Server in the database system serves as a persistent storage layer, responsible for creating the "OCRdata" database and tables containing fields such as picture_number (primary key), picture_path, context1-context4, created_at, etc. It has the ability to dynamically detect and automatically create tables and fields. It initializes the connection through a connection string and monitors the status. When data is inserted, it is written into the table through SQL statements. If the insertion fails, it will be rolled back and the Redis cache will be deleted to ensure consistency. When querying, Redis is supplemented with missing data. When reading, full data is loaded into Redis to ensure the reliability and integrity of long-term data storage.

[0067] Further, the step S5 specifically includes the following sub-steps:

[0068] S51, the parameter setting module sets:

[0069] This module is used to configure the basic parameters related to system operation. Through QLineEdit, QComboBox, QSpinBox, etc. Form controls, it realizes the initialization setting of key running parameters such as recognition interval, serial number, baud rate, RTSP address, video frame rate, model path, template loading path, etc. All settings are real-time bound to the global configuration manager, supporting dynamic modification and automatic persistence as configuration files (such as INI or YAML format).

[0070] S52, the system control module is set: the module provides visual control interface of system running state, including start / pause identification, template switching, task resetting, device connection test and other operation functions. QPushButton, QStatusBar and QTimer are used to realize periodic task driving and state echo, and the interface synchronously displays real-time information such as current running state, thread state and identification count statistics of the system.

[0071] S53, the virtual keyboard and mouse control module is set:

[0072] The operation template for configuring and managing virtual mouse and keyboard is set. QTreeView and QStandardItemModel are used to display the operation sequence list, support adding, deleting, modifying and querying operation instructions, and the content includes operation type (click, scroll), execution coordinate, key value, delay time and trigger condition fields. The template data is read and written in JSON format, and is linked with the actual task state, can be transmitted to the bottom layer Shineblink hardware development board to execute physical simulation operation in real time.

[0073] S54, the template matching establishment module is set:

[0074] The module is used to build and edit feature value template and content recognition template. QGraphicsView is used to realize image display and ROI frame selection, and users can manually frame the recognition area in the specified image area, input field name, type, post-processing rule and other metadata. The matching template supports visual comparison preview, confidence setting and priority adjustment, what you see is what you get, which facilitates engineers to quickly build multi-interface recognition configuration system. At the same time, the picture preprocessing mode template is configured, which improves the recognition rate.

[0075] S55, the video stream display module is set:

[0076] OpenCV and QThread are integrated to realize asynchronous decoding and image buffering of RTSP video stream, and QLabel+QPixmap is used to update the picture in real time, and the frame rate is controllable. Supports screenshot, picture zooming and recognition area superimposed display function, provides visual reference basis for subsequent image recognition and template labeling.

[0077] S56, the log monitoring module is set;

[0078] QTextEdit control is used to build multi-level log terminal, supports INFO / WARNING / ERROR hierarchical output, and outputs log records combined with time stamp and thread label. Supports log scrolling, filtering, automatic cleaning and exporting functions, which is convenient for fault backtracking and system maintenance. Log data and operation records can be automatically associated and uploaded to the database for structured storage.

[0079] Compared with the prior art, the application has the advantages that:

[0080] 1. The application adopts a non-invasive video stream acquisition technology, obtains image data through a device native HDMI interface, avoids modification of a device system, is not limited by communication protocols and interface standards, can be widely applied to various old or closed protocol industrial devices, and has strong compatibility and promotion.

[0081] 2. The application introduces a virtual mouse control module, realizes remote clicking and automatic polling of a device interface in combination with an RPA technology, effectively solves the problem of multi-page parameter reading in combination with an image recognition process, and significantly improves data acquisition coverage and automation level of the system.

[0082] 3. The application constructs a parameter-adjustable and process-visualized user interface, allows a user to freely configure recognition time, screenshot interval, template file and other running parameters, and enhances interactive friendliness and engineering adaptability of the system.

[0083] 4. The whole architecture of the application adopts wired connection throughout, data transmission is stable and reliable, in combination with a multi-threaded task scheduling mechanism, 24-hour continuous operation can be realized, and the application has high stability and actual deployment value in an industrial scene.

[0084] Comparison of the application with the prior art and innovation point explanation

[0085] The core of the application is a non-invasive device monitoring system design method based on an OCR technology, which realizes automatic and accurate extraction of industrial device interface data in combination with video stream acquisition, virtual keyboard and mouse control, dynamic template matching, a PP-OCRv4 model of PaddleOCR and Redis database and SQLServer database storage. The following compares the application with a traditional OCR scheme and a communication protocol acquisition scheme to clearly explain the innovation points and differences of the application.

[0086] I. Comparison with the traditional OCR scheme

[0087] The traditional OCR scheme is mainly applied to text recognition of static images in an industrial scene, and is usually used for document scanning or simple screenshot processing. The following is the difference between the application and the traditional OCR scheme.

[0088]

[0089] Innovation points:

[0090] 1. Dynamic template matching mechanism: through a double-layer mechanism of a feature value template and a content recognition template, the system can automatically identify an interface type and switch to a corresponding template, and solve the deficiency of the traditional OCR scheme in multi-interface adaptation.

[0091] 2. Virtual keyboard and mouse automation: Introduce Shineblink hardware development board and keyboard and mouse simulator CH9329 to simulate real mouse and keyboard operation, realize interface automatic interaction, and significantly improve the automation degree of data collection.

[0092] 3. Industrial scene optimization: The PP-OCRv4 model is optimized for industrial complex environment, combined with configurable preprocessing template, which enhances the robustness to low quality images (such as blur, noise).

[0093] 4. Modularity and visualization: Through PySide6 host computer and Redis database, a highly integrated and visual monitoring system is built, which is convenient for user operation and data management.

[0094] II. Comparison with communication protocol collection scheme

[0095]

[0096] Innovation points:

[0097] 1. Non-invasive collection: Through HDMI video stream and virtual keyboard and mouse operation, it does not need to rely on device communication protocol, avoids hardware modification or protocol adaptation, and is suitable for old or closed systems.

[0098] 2. Wide coverage data collection: Unstructured data (such as state parameters, alarm information) displayed on the interface can be extracted, which makes up for the deficiency of communication protocol scheme that can only collect structured data.

[0099] 3. Fast deployment and expansion: Template design and general hardware module (HDMI encoder + USB control) reduce deployment threshold and support rapid adaptation to new devices.

[0100] 4. High stability: wired connection and multi-thread scheduling mechanism ensure long-term stable operation of the system, meeting the high reliability requirements of industrial scenes. BRIEF DESCRIPTION OF DRAWINGS

[0101] Figure 1 The overall steps of the present application are shown in the figure.

[0102] Figure 2 The database hierarchical data management diagram is shown in the figure.

[0103] Figure 3 The video collection hardware connection diagram of the present application is shown in the figure.

[0104] Figure 4 The virtual mouse hardware connection diagram of the present application is shown in the figure.

[0105] Figure 5 The virtual keyboard and mouse custom interface diagram of the present application is shown in the figure.

[0106] Figure 6 Interface diagram for customizing the recognition template of the present application.

[0107] Figure 7 Interface indication diagram for the device monitoring system of the present application. DETAILED DESCRIPTION

[0108] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the protection scope of the present application.

[0109] As shown in Figure 1 The specific implementation steps of the design method of the non-invasive device monitoring system based on optical character recognition (OCR) of the present application are as follows:

[0110] S1, use a video encoder and a switch to encode and output a video source of a semiconductor or photovoltaic device, input into a local industrial personal computer (IPC) for decoding processing, connect the IPC and the semiconductor or photovoltaic device through a USB serial port based on a hardware development board and a keyboard and mouse simulator (CH9329), realize remote keyboard and mouse control of the IPC on the semiconductor or photovoltaic device, and finally build an overall hardware connection structure.

[0111] S2, establish a template matching mechanism, set characteristic values and content recognition templates for different interfaces of the semiconductor or photovoltaic device, establish a virtual keyboard and mouse operation template, and establish a picture preprocessing mode template.

[0112] S3, use a PaddleOCR framework, select a PP-OCRV4 model as a main model to recognize and process each image that needs to be recognized.

[0113] S4, use a hybrid architecture combining a relational database (SQL Server) and an in-memory data storage (Redis) to store data.

[0114] S5, build an overall IPC interface based on a Pyside6 framework to monitor and control the recognition process.

[0115] In this embodiment, the overall platform of the optical character recognition (OCR) based non-invasive equipment monitoring system is developed using Python language, realizing modular and layered architecture design. The system mainly includes six functional parts: image acquisition module, virtual keyboard and mouse interaction module, template calibration module, OCR recognition module, data storage module and user visual interface module. The core OCR engine of the system selects PP-OCRv4 version in PaddleOCR open source framework. This model has the advantages of fast recognition speed, strong adaptability and high accuracy, and is especially suitable for video image processing tasks in complex industrial environments.

[0116] Figure 2 To balance the efficient access of recognition data and the performance of system running, the system adopts a layered data management structure combining Redis and SQL Server. After completing the OCR recognition, the system first writes the recognition results (including picture number, recognition field, timestamp, etc.) in the form of key-value pairs into the in-memory database Redis. As a high-performance cache layer, Redis supports millisecond-level writing and multi-thread concurrent access, significantly improving the system's data processing speed and running smoothness. The system background scheduling task will synchronize the recognition results in Redis to the SQL Server database according to the set period or data change trigger mechanism, realizing reliable and persistent storage of data. SQL Server, as a structured relational database, is responsible for archiving, classifying and indexing long-period data, ensuring data integrity and traceability. When the user initiates a historical data query request through the industrial computer interface, the system will extract all corresponding data from SQL Server at once and write it into Redis cache. The subsequent historical record query, data filtering and display operations in the interface are completed by accessing Redis, avoiding frequent calls to SQL database, effectively improving data response speed and reducing database pressure.

[0117] Figure 3 The schematic diagram of video capture hardware connection is given. The image acquisition module connects the video output end (HDMI interface) of the automation equipment, uses a high-definition encoder to compress the real-time video stream into H.264 format, and pushes it to the processing end based on RTSP protocol in the local area network through the switch. The processing end uses OpenCV library to perform real-time analysis and key frame extraction on the video stream, and automatically intercepts the image frames that need to be recognized. To adapt to different states and layouts of device screen display, the system is equipped with image preprocessing function, which can perform grayscale, binarization, noise removal, image enhancement and other operations on the image, thereby improving the OCR recognition accuracy.

[0118] Figure 4The virtual mouse hardware connection diagram is given, and the virtual mouse interaction module is realized based on the ShineBlink industrial control development board and the CH9329 keyboard and mouse simulator, which can simulate the mouse behavior of standard USB HID devices. The control end issues coordinate commands to the development board through TCP communication, and the development board converts them into mouse click signals to complete the point-and-click operation on the automation device interface, realizing interactive operations such as page turning and menu switching. The module supports two running modes: "automatic polling" and "manual interaction", which can be flexibly selected according to the on-site needs in actual deployment.

[0119] Figure 5 and Figure 6 The virtual keyboard and mouse custom interface diagram and the recognition template custom interface diagram are given, and the template calibration module includes virtual mouse template customization and recognition template customization. Recognition template customization includes feature value template establishment, content recognition template establishment, and image preprocessing method establishment. The template calibration module provides graphical interactive tools, allowing users to demarcate each device interface and generate corresponding template files (JSON format) for subsequent image recognition to quickly locate the target data area. Each template corresponds to a specific device interface, and the template is automatically matched by the feature value in the image, realizing the recognition and automatic switching of multiple devices and multiple pages.

[0120] In the OCR recognition process, the system extracts the region of interest according to the recognition template, inputs it one by one into the PP-OCRv4 model for character recognition, and converts the recognition results into structured data according to the set field mapping. After format conversion, the recognition data is stored in a hybrid architecture combining relational databases (SQL Server) and in-memory data storage (Redis). The system database design supports field indexing, timestamp marking, and device number classification, facilitating later query, analysis, and traceability. At the same time, the system retains the original image data and associates the recognition results to achieve one-to-one correspondence between data and images, enhancing the data credibility and auditability of the system.

[0121] Figure 7 The interface indication diagram of the device monitoring system is given. The semiconductor / photovoltaic device monitoring system uses PySide6 to build a cross-platform GUI interactive interface, which mainly consists of a switching interface area (1), a control area (2), a parameter setting display area (3 and 4), and a real-time result display area (5). The parameter setting display area consists of a basic parameter setting display area (3) and a virtual mouse area (4). It supports recognition parameter configuration, template selection, recognition process monitoring, real-time video stream display, and abnormal alarm pop-up window. The system supports flexible configuration of parameters such as screenshot interval, recognition period, and total running time, and displays recognition progress, current recognition content, and database upload status, improving user operation convenience and system transparency.

[0122] To verify the stability and recognition performance of the system, the complete system is deployed in an experimental environment and continuously run test. The test duration is 24 hours, the system completes multiple rounds of data acquisition and recognition tasks in uninterrupted running state, the actual measurement OCR recognition accuracy reaches 99.2%, the data upload success rate reaches 100%, the image acquisition success rate reaches 100%, the OCR accuracy fluctuation is within 0.03%, which meets the demand of high precision and low error industrial data recognition. At the same time, the system supports concurrent monitoring request of not less than 100 sets of equipment, the response delay is less than 1 second, which fully verifies the parallel processing capability and deployment reliability of the system.

[0123] In summary, the non-invasive equipment monitoring system method based on optical character recognition OCR provided by the application integrates multiple key technologies such as image acquisition, interactive control, template management, character recognition and data processing, has many advantages such as non-invasive acquisition, high recognition rate, real-time response, stable operation, strong expansibility, etc., can effectively solve the problem that traditional industrial equipment cannot realize digital monitoring due to communication protocol limitation, provides a unified, visual and intelligent data acquisition scheme for heterogeneous industrial systems, and significantly improves the operation and maintenance efficiency and digital level of the production site.

[0124] The above embodiments are only preferred implementation examples of the application and should not be understood as limiting the protection scope of the application. Any equivalent modification, structure optimization and technical replacement made within the spirit and technical principles of the application should be covered within the protection scope of the application.

Claims

1. A design method for a non-intrusive device monitoring system based on optical character recognition (OCR), characterized in that, Includes the following steps: S1 uses a video encoder and switch to encode and output the video source of the device, and inputs it to the local industrial control computer for decoding. Based on the hardware development board and keyboard and mouse simulator, the industrial control computer and the device are connected to realize the remote keyboard and mouse control of the device by the industrial control computer, and finally the overall hardware connection structure is built. S2, establish a template matching mechanism, set feature value templates and content recognition templates for different interfaces of the device, establish virtual keyboard and mouse operation templates, and establish image preprocessing method templates; S3 uses the PaddleOCR framework, selects the PP-OCRV4 model for OCR processing, and employs the text detection algorithm DB and the text recognition algorithm SVTR. S4 uses a hybrid architecture that combines relational database SQL Server with in-memory data storage Redis for data storage; S5 uses the Pyside6 framework to build an overall industrial control computer interface to monitor and control the recognition process.

2. The design method of a non-intrusive device monitoring system based on optical character recognition (OCR) according to claim 1, characterized in that, Step S1 specifically includes the following sub-steps: S11, Hardware setup for video streaming transmission: The system's video acquisition component includes an HDMI video encoder, an Ethernet switch, network cables, and an industrial control computer. The HDMI video encoder converts the device's video signal into an H.264 format video stream and pushes it to the network via the RTSP protocol. The encoded video stream enters the Ethernet switch via a network cable and is forwarded to the industrial control host according to the MAC address; the industrial control host decodes the video stream, generates continuous image frames, and hands them over to the OCR recognition module for processing, realizing the extraction, analysis and storage of key information; S12, hardware setup for virtual keyboard and mouse: The hardware consists of a Shineblink hardware development board and a CH9329 keyboard and mouse simulator. It establishes a communication link with an industrial control computer via a USB serial port. The Shineblink hardware development board acts as the main control unit, receiving software instructions from the industrial control computer and driving the CH9329 keyboard and mouse simulator to simulate real mouse or keyboard operations after parsing them through the serial communication protocol.

3. The design method of a non-intrusive device monitoring system based on optical character recognition (OCR) according to claim 2, characterized in that, Step S2 specifically includes the following sub-steps: S21, Establish feature value templates and content recognition templates: The two-layer template matching mechanism consists of feature value templates and content recognition templates. To identify character information, the system first calls the feature value template to extract feature values ​​from a preset area in the image, and then compares the current feature with the established feature template library to identify the current page type and switch to the corresponding content recognition template in the content recognition template library. The content recognition template then segments the entire image based on the coordinate information in the template before identification. To establish feature template library and content recognition template library, the system designed an OCR template creation program interface based on PySide6. On this interface, users can freely select the coordinate regions of interest on the image and name and save them. All template files adopt the structured JSON file format for the OCR recognition module to read. S22, Create a virtual keyboard and mouse operation template: Design a virtual keyboard and mouse operation template mechanism to control various interface interaction behaviors at the command level; S23, Create an image preprocessing template: To further improve recognition accuracy and template adaptation, a matching image preprocessing template mechanism has been established. The system allows users to specify an image preprocessing template mechanism for each OCR recognition module during the template configuration process. The image preprocessing template supports the combined use of the following processing modules: image grayscale conversion, adaptive binarization, image enhancement, color inversion, edge enhancement, noise reduction filtering, rotation correction, and scale normalization. Each processing operation can be configured with its parameter values, and the system will execute them step by step in the order defined in the template before the OCR recognition module recognizes the image.

4. A design method for a non-intrusive device monitoring system based on optical character recognition (OCR) according to claim 1, 2, or 3, characterized in that, Furthermore, step S4 specifically includes the following sub-steps: S41, the relational database SQL Server, is responsible for persistent data storage, ensuring data reliability and integrity: SQL Server, a relational database in the database system, serves as the persistent storage layer. It is responsible for creating the OCRdata database and tables containing fields such as picture_number, picture_path, context1-context4, and created_at. It has the ability to dynamically detect and automatically create tables and fields. It initializes the connection through the connection string and monitors the status. When data is inserted, it is written to the table via SQL statements. If the data fails, it rolls back and deletes the Redis cache to ensure consistency. When querying, it supplements the missing data in Redis. When reading, it loads the full data into Redis to ensure the reliability and integrity of long-term data storage. S42 uses Redis as a high-speed caching layer for in-memory data storage, optimizing data access speed and improving system response performance.

5. The design method of a non-intrusive device monitoring system based on optical character recognition (OCR) according to claim 4, characterized in that, Step S5 specifically includes the following sub-steps: S51, parameter setting module settings: This module is used to configure basic parameters related to system operation. It uses QLineEdit, QComboBox, and QSpinBox form controls to initialize key operating parameters such as recognition interval, serial port number, baud rate, RTSP address, video frame rate, model path, and template loading path. All settings are bound to the global configuration manager in real time, supporting dynamic modification and automatic persistence to configuration files. S52, System Control Module Settings: This module provides a visual control interface for the system's running status, including start / pause identification, template switching, task reset, and device connection test operation functions. It uses QPushButton, QStatusBar, and QTimer to implement periodic task-driven and status feedback. The interface synchronously displays the system's current running status, thread status, and real-time identification count statistics. S53, Virtual Keyboard and Mouse Control Module Settings: It is used to configure and manage operation templates for virtual mouse and keyboard. The operation sequence list is displayed through QTreeView and QStandardItemModel. It supports adding, deleting, modifying and querying operation commands. The content includes operation type, execution coordinates, key value, delay time and trigger condition fields. Template data is read and written in JSON format and linked with the actual task status. It is scheduled and transmitted to the underlying Shineblink control module in real time to execute physical simulation operations. S54, Template Matching Module Settings: This module is used to build and edit OCR region recognition templates and interface feature matching templates. It uses QGraphicsView to display images and select ROIs. Users can manually select recognition areas in a specified image area and input field names, types, post-processing rules, and metadata. The matching templates support visual comparison previews, confidence settings, and priority adjustments, and also configure image preprocessing templates. S55, Video Stream Display Module Settings: It integrates OpenCV and QThread to achieve asynchronous decoding and image buffering of RTSP video streams, and uses QLabel+QPixmap to update the screen in real time with controllable frame rate; it supports screenshot, screen zoom and recognition area overlay display functions to provide visual reference for subsequent image recognition and template annotation. S56, Log Monitoring Module Settings: A multi-level log terminal is built using the QTextEdit control, which supports INFO / WARNING / ERROR hierarchical output. Log records are output with timestamps and thread tags. Log scrolling, filtering, automatic clearing and export functions are supported, which facilitates fault traceability and system maintenance. Log data and operation records are automatically associated and uploaded to the database for structured storage.

Images