Transformer substation monitoring interface intelligent verification method and system based on multi-modal data fusion

Abstract

The invention provides a substation monitoring interface intelligent verification method and system based on multi-modal data fusion. The method specifically comprises the following steps: S1, simulating server point location transformation operation; s2, the client obtains a point position change image based on the monitoring picture; the obtained image is input into a trained target detection model to obtain a target detection result, the target detection result comprises a target bounding box, a category and confidence, the target bounding box is a bright light word plate position with a device name, and the category is the device name; screenshot is carried out according to the target bounding box; s3, importing the screenshot into the trained character recognition model, and performing recognition and extraction of natural language characters; and S4, normalizing the device name obtained in the S2 and the natural language characters extracted in the S3, and outputting a text file for a tester to refer to and use.

Description

Technical Field

[0001] The present invention belongs to the technical field of intelligent operation and maintenance of power systems, and specifically relates to a method and system for intelligent verification of a substation monitoring interface based on multimodal data fusion. Background Art

[0002] In the current trend toward automation and intelligent power generation and transportation, technologies such as image and text recognition and AI-powered language model-assisted question answering or generation have gradually become the hallmarks of high-tech products. To adapt to the future development of substations, State Grid Corporation of China proposed in 2020 to optimize local substation operation and maintenance models based on local conditions and implement centralized substation monitoring. In response to State Grid Corporation's call, the need for professional testing of the functional applications of centralized control station equipment monitoring systems has emerged. Testing organizations not only need to simultaneously test the systems under test from multiple test institutions, but also need to establish a large network communication environment at the station control level during the testing process, increasing time and labor costs. Acceptance agencies, on the other hand, face the challenges of tight deadlines and short acceptance times associated with replacing new systems. Existing technical solutions have three limitations: 1) Static configuration modes cannot adapt to the real-time verification requirements of dynamic image elements; 2) Traditional optical character recognition (OCR) technology has insufficient accuracy in multi-layer semantic parsing of composite light-emitting signboards; and 3) The lack of a multi-dimensional semantic mapping mechanism for SCD models results in a lack of traceability in test reports. Especially during the acceptance process of the new generation of centralized control station equipment monitoring system, the traditional manual comparison method has hidden error detection blind spots. Summary of the Invention

[0003] The purpose of the present invention is to propose an intelligent verification method and system for substation monitoring interface based on multimodal data fusion. By partitioning and identifying each module and continuously refreshing the description of the light-emitting sign, it is possible to quickly identify the flashing behavior based on the input video image signal, thereby realizing real-time capture of the expression content of all monitoring graphics, and finally dynamically binding with the telemetry, telesignaling and remote control points in the SCD model file, and providing a phased test report and error correction suggestions for the correspondence between the monitoring graphic interface and the SCD model file.

[0004] To achieve the above object, the technical solution of the present invention is as follows: The present invention proposes an intelligent verification method for substation monitoring interface based on multimodal data fusion, which specifically includes the following steps: Step S1, simulating the server point change operation; Step S2: The client obtains a point change image based on the monitoring screen; inputs the obtained image into the trained target detection model to obtain a target detection result, including a target bounding box, a category, and a confidence level. The target bounding box is the location of the lighted sign with the device name, and the category is the device name; and a screenshot is taken based on the target bounding box. Step S3: import the screenshot into the trained text recognition model to perform natural language text recognition and extraction; Step S4: normalize the device name obtained in step S2 and the natural language text extracted in step S3, and output them as a text file for reference by testers.

[0005] Preferably, the acquiring of the point position change image is specifically: using a camera to continuously take screenshots of the target area at the same frequency as the video signal refresh frequency.

[0006] Preferably, the training of the target detection model specifically includes: S21. Manually set all possible indicator light combinations, and capture and save corresponding images based on the monitoring screen; S22, manually labeling the target bounding box and category of the image, and performing image preprocessing; S23. Use the image processed in step S22 to train the target detection model to obtain a trained target detection model.

[0007] Preferably, the image preprocessing includes: Perform image enhancement operations to increase the diversity of the dataset, including rotating, scaling, cropping, adjusting brightness and contrast; Normalization and resizing: Normalize image data to a uniform size and resize the image to fit the model input.

[0008] Preferably, the target detection model adopts the YOLOv8 model, and uses cross entropy loss and IoU loss as loss functions for model training.

[0009] Preferably, in step S2, the acquired image is input into a trained target detection model to obtain a target detection result as follows: the YOLO model divides the input image into multiple grids and predicts the target bounding box and category probability in each grid; and removes redundant bounding boxes through non-maximum suppression, retaining the detection result with the highest confidence, and drawing the detected target bounding box, category and confidence on the image according to the detection result, and then outputting it into a picture.

[0010] Preferably, if the confidence level in the detection result is greater than 90%, then the association in the current monitoring image is considered to be correct, otherwise the association is incorrect.

[0011] Preferably, the step of importing the screenshot into a trained text recognition model to perform natural language text recognition and extraction specifically includes: S31, pre-processing the captured image; S32, performing text region monitoring on the pre-processed image, using edge detection and connected domain analysis algorithms to detect text regions in the image and generate text bounding boxes; and then segmenting the detected text regions into separate text images; S33: Input the segmented text image into a text recognition model for text recognition, convert the text in the image into corresponding character codes, and output it in text format.

[0012] Preferably, step S31 specifically includes: Grayscale: Convert a color image to a grayscale image; Binarization: convert grayscale image into black and white binary image; Denoising: remove noise points in the image; Normalization: Adjust the size and scale of the image to make it meet the model input requirements; Tilt Correction: Make the text aligned horizontally.

[0013] The present invention also proposes an intelligent verification system for a substation monitoring interface based on multimodal data fusion, which includes a processor, a memory, and a computer program stored on the memory. When the processor executes the computer program, it specifically executes any step in the above-mentioned intelligent verification method for the substation monitoring interface.

[0014] Compared with the prior art, the present invention has the following beneficial effects: The present invention reduces the workload of point-to-point debugging of substations after technical transformation to a large extent, and can be combined with a tester to realize a fully automatic process of remote measurement, remote signaling and remote control point-to-point solution. It is suitable for monitoring equipment in substation monitoring systems that use most image information as display media, can effectively support the smooth development of substation technical transformation work, and has good engineering application value and prospects.

[0015] Field tests have verified that this solution offers three major breakthroughs compared to traditional methods: 1) Significantly improved acceptance efficiency, achieving consistency in detection speed and icon refresh speed; 2) Industry-leading semantic parsing accuracy; and 3) Traceability facilitates the subsequent automatic generation of intelligent diagnostic reports including error path analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 This is an architectural diagram of data transmission between the substation monitoring system and the operator display page of the present invention; Figure 2 This is a test flow chart of the intelligent parsing and processing of substation data of the present invention; Figure 3 This is a flow chart of the target detection model training process of the present invention; Figure 4 This is the flow chart of the text recognition model reasoning of the present invention. DETAILED DESCRIPTION

[0017] The following is combined with Figure 1-4 , the technical solution of the present invention is described in detail.

[0018] This paper proposes an intelligent verification method for substation monitoring interfaces based on multimodal data fusion. For 27 optical signboards in the SCD file (25 of which can be associated with specific points in the SCD), 27 batches of automatic testing pipelines are simultaneously formed. The specific content of each batch includes the following steps: Step S1: CMS / MMS simulates the server position change operation, including position change (for remote signaling, such as light-emitting signs, switches, pressure plates, etc., generally called position change) and value setting (for analog quantities, such as voltage and current, generally called value setting) operations; Step S2: The client obtains a point change image based on the monitoring screen; inputs the obtained image into the trained target detection model to obtain a target detection result, including a target bounding box, a category, and a confidence level. The target bounding box is the location of the lighted sign with the device name, and the category is the device name; and a screenshot is taken based on the target bounding box. Step S3: import the screenshot into the trained text recognition model to perform natural language text recognition and extraction; Step S4: normalize the device name obtained in step S2 and the natural language text extracted in step S3, and output them as a text file for reference by testers.

[0019] After 27 batches of tests are completed, the test results will be output to a file in the prescribed format for archiving and reference and use by testers.

[0020] In this embodiment, the step of acquiring the point change image specifically includes: using a camera to continuously take screenshots of the target area at a frequency that is the same as the refresh frequency of the video signal.

[0021] In this embodiment, the training of the target detection model specifically includes: S21. Manually set all possible indicator light combinations, and capture and save corresponding images based on the monitoring screen; S22, manually labeling the image with a target bounding box (the location of the lighted sign with the device name) and a category (device name), and performing image preprocessing; S23. Use the image processed in step S22 to train the target detection model to obtain a trained target detection model.

[0022] In this embodiment, the image preprocessing includes: Perform image enhancement operations to increase the diversity of the dataset, including rotating, scaling, cropping, adjusting brightness and contrast; Normalization and resizing: Normalize image data to a uniform size and resize the image to fit the model input.

[0023] In this embodiment, the object detection model uses the YOLOv8 model, and uses cross-entropy loss and IoU loss as loss functions to calculate model performance. Model parameters are updated through backpropagation. Model performance can be evaluated using metrics such as accuracy, recall, and mAP (mean Average Precision).

[0024] In this embodiment, in step S2, the acquired image is input into the trained target detection model to obtain the target detection result. Specifically, the YOLO model divides the input image into multiple grids and predicts the target bounding box and category probability in each grid; and removes redundant bounding boxes through non-maximum suppression, retaining the detection result with the highest confidence, and draws the detected target bounding box, category and confidence on the image according to the detection result, and then outputs it as a picture.

[0025] In this embodiment, if the confidence level in the detection result is greater than 90%, it is considered that the association in the current monitoring image is correct, otherwise the association is incorrect.

[0026] In this embodiment, the step of importing the screenshot into a trained text recognition model to perform natural language text recognition and extraction specifically includes: S31, pre-processing the captured image; S32. Detect text regions in the preprocessed image using edge detection and connected domain analysis algorithms to detect text regions in the image and generate text bounding boxes; then segment the detected text regions into separate text images (units such as words, characters, etc.); S33: Input the segmented text image into a text recognition model for text recognition, convert the text in the image into corresponding character codes, and output it in text format.

[0027] The text recognition model first extracts features from the segmented text image. These features include key information such as strokes, shape, and texture. These extracted features are then matched against a pre-trained text recognition model. This model uses a convolutional neural network, trained on a large number of text samples, to recognize text in various fonts and styles. Finally, the model outputs the recognition result, converting the text in the image into its corresponding character encoding and outputting it in text format for subsequent use.

[0028] The above process is a conversion process from image to text, which achieves efficient and accurate text extraction through steps such as image preprocessing, text detection, text recognition, and natural language processing.

[0029] In this embodiment, step S31 specifically includes: Grayscale: Convert color images into grayscale images to reduce the amount of data while removing the interference of color information; Binarization: Convert grayscale images into black and white binary images for easy subsequent processing; Denoising: remove noise points in the image to improve image quality; Normalization: Adjust the size and scale of the image to make it meet the model input requirements; Tilt correction: If the text in the image is tilted, it needs to be corrected so that the text is aligned horizontally.

[0030] The present invention also proposes an intelligent verification system for a substation monitoring interface based on multimodal data fusion, which includes a processor, a memory, and a computer program stored on the memory. When the processor executes the computer program, it specifically executes any step in the above-mentioned intelligent verification method for the substation monitoring interface.

[0031] In summary, the present invention innovatively proposes a three-level intelligent verification architecture: 1. Visual perception layer: Due to the low difficulty of task target detection, a single-stage algorithm that directly predicts bounding boxes and categories is used to extract the high-frequency refresh signal features of the light-emitting sign state changes and light-on signals; 2. Semantic parsing layer: Build a deep OCR model optimized for knowledge in the power sector, especially for the use of photonic signs, to support multi-level semantic deconstruction of composite photonic signs; 3. Model coupling layer: Combining optical character recognition and target detection algorithms, it can accurately and concisely analyze and transmit the real-time changing information in the video transmission signal to the substation operator's display page, and automatically edit alarm conditions to reduce the operator's burden.

[0032] The testing method proposed in the present invention can be used in a variety of application scenarios: first, it can be used for functional testing in the laboratory of the Institute of Electrical Engineering, and laboratory functional testing can be carried out on the centralized control system based on the domestic communication protocol, which can reduce the repeated and redundant operations of the testers and improve the testing efficiency; second, it can be used for factory acceptance of equipment (system) manufacturers. Before the equipment monitoring system of the centralized control station leaves the factory, simulation testing is carried out on equipment that does not yet have a debugging environment according to the actual project site configuration, so as to advance the debugging acceptance work and discover functional and parameter configuration defects in advance; third, it can be used for project site debugging and acceptance. During the project site debugging stage, this test method is used to verify the online monitoring data of the primary and secondary equipment, check the measurement and alarm data of the auxiliary equipment, confirm the control operation instructions of the auxiliary equipment, shorten the on-site debugging cycle, and reduce the power outage time.

[0033] The above are preferred embodiments of the present invention. Any changes made according to the technical solution of the present invention, as long as the resulting functions and effects do not exceed the scope of the technical solution of the present invention, shall fall within the scope of protection of the present invention.

Claims

1. An intelligent verification method for substation monitoring interface based on multimodal data fusion, characterized in that: The specific steps include: Step S1, simulating the server point change operation; Step S2: The client obtains a point change image based on the monitoring screen; inputs the obtained image into the trained target detection model to obtain a target detection result, including a target bounding box, a category, and a confidence level. The target bounding box is the location of the lighted sign with the device name, and the category is the device name. And take screenshots based on the target bounding box; Step S3: import the screenshot into the trained text recognition model to perform natural language text recognition and extraction; Step S4: normalize the device name obtained in step S2 and the natural language text extracted in step S3, and output them as a text file for reference by testers.

2. The intelligent verification method for substation monitoring interface based on multimodal data fusion according to claim 1 is characterized in that: The method of obtaining the point change image specifically includes: using a camera to continuously take screenshots of the target area at a frequency that is the same as the refresh frequency of the video signal.

3. The intelligent verification method for substation monitoring interface based on multimodal data fusion according to claim 1 is characterized in that: The training of the target detection model specifically includes: S21. Manually set all possible indicator light combinations, and capture and save corresponding images based on the monitoring screen; S22, manually labeling the target bounding box and category of the image, and performing image preprocessing; S23. Use the image processed in step S22 to train the target detection model to obtain a trained target detection model.

4. The intelligent verification method for substation monitoring interface based on multimodal data fusion according to claim 3 is characterized in that: The image preprocessing includes: Perform image enhancement operations to increase the diversity of the dataset, including rotating, scaling, cropping, adjusting brightness and contrast; Normalization and resizing: Normalize image data to a uniform size and resize the image to fit the model input.

5. The intelligent verification method for substation monitoring interface based on multimodal data fusion according to claim 3 is characterized in that: The target detection model adopts the YOLOv8 model, and uses cross entropy loss and IoU loss as loss functions for model training.

6. The intelligent verification method for substation monitoring interface based on multimodal data fusion according to claim 5 is characterized in that: In step S2, the acquired image is input into a trained target detection model to obtain a target detection result. Specifically, the YOLO model divides the input image into multiple grids and predicts the target bounding box and category probability in each grid; and removes redundant bounding boxes through non-maximum suppression, retaining the detection result with the highest confidence, and draws the detected target bounding box, category and confidence on the image according to the detection result, and then outputs it as a map.

7. The intelligent verification method for substation monitoring interface based on multimodal data fusion according to claim 6 is characterized in that: If the confidence level in the detection result is >90%, the association in the current monitoring image is considered correct, otherwise the association is incorrect.

8. The intelligent verification method for substation monitoring interface based on multimodal data fusion according to claim 1 is characterized in that: The step of importing the screenshot into a trained text recognition model to perform natural language text recognition and extraction specifically includes: S31, pre-processing the captured image; S32, performing text region monitoring on the pre-processed image, using edge detection and connected domain analysis algorithms to detect text regions in the image and generate text bounding boxes; and then segmenting the detected text regions into separate text images; S33: Input the segmented text image into a text recognition model for text recognition, convert the text in the image into corresponding character codes, and output it in text format.

9. The intelligent verification method for substation monitoring interface based on multimodal data fusion according to claim 8 is characterized in that: Step S31 specifically includes: Grayscale: Convert a color image to a grayscale image; Binarization: convert grayscale image into black and white binary image; Denoising: remove noise points in the image; Normalization: Adjust the size and scale of the image to make it meet the model input requirements; Tilt Correction: Make the text aligned horizontally.

10. The intelligent verification system for substation monitoring interface based on multimodal data fusion is characterized by: The method comprises a processor, a memory and a computer program stored in the memory. When the processor executes the computer program, the method specifically performs the steps of the intelligent verification method for the substation monitoring interface as described in any one of claims 1 to 9.

Images