Substation pointer meter identification method and system based on multi-modal large model

Abstract

The invention discloses a transformer substation pointer meter identification method and system based on a multi-modal large model, and the method comprises the steps: constructing and training a pointer meter region target detection model, and carrying out the fine adjustment of the multi-modal large model, and forming a meter multi-modal model. An inspection device monitors and collects a to-be-recognized pointer meter image and transmits the to-be-recognized pointer meter image to the meter area target detection model and the meter multi-mode model in sequence, and finally the pointer meter reading is obtained. According to the invention, the multi-mode large model is matched with the meter image text to carry out fine tuning improvement on the data set, and the target detection model is combined to realize the monitoring and alarm of the reading of each pointer meter in the transformer substation, so that the problems of high manpower inspection consumption, low efficiency, recording errors and the like are avoided.

Description

Technical Field

[0001] The present invention relates to the field of computer vision and multimodal large models, and in particular to a substation pointer meter recognition method and system based on a multimodal large model. Background Art

[0002] With the continuous development of power systems and the improvement of automation levels, substations, as the core components of power systems, are crucial to ensuring grid stability and power supply quality. The traditional method of identifying pointer meters in substations mainly relies on inspection personnel’s naked eye identification and manual registration, which has problems such as low efficiency, high cost and high risk.

[0003] Therefore, there is an urgent need for a more efficient intelligent identification technology for substation pointer meters. Summary of the invention

[0004] In order to make up for the shortcomings of the existing technical problems, the purpose of the present invention is to provide a substation pointer meter identification method and system based on a multimodal large model, identify the pointer meter area based on a lightweight target detection network model, expand and crop the pointer meter, and then use the multimodal large model to perform question and answer recognition detection to obtain the reading of the pointer meter.

[0005] The present invention discloses a method for identifying pointer meters in a substation based on a multi-modal large model, the steps comprising:

[0006] The inspection equipment collects original images of various types of pointer meters in the substation, and constructs a meter area detection data set based on the original images; establishes a target detection model, and performs supervised learning training on the target detection model through the meter area detection data set to obtain a meter area target detection model;

[0007] Performing expansion-cropping processing on the pointer meter image, providing a text description for the pointer meter image after the expansion-cropping processing, extracting prompt words, and constructing a meter image-text pair data set; establishing a multimodal large model, and performing supervised learning training on the multimodal large model through the meter image-text pair data set to obtain a meter multimodal model;

[0008] The inspection equipment collects the image of the pointer meter to be identified, and inputs the image of the pointer meter to be identified into the meter area target detection model to obtain the meter area detection result; the image of the pointer meter to be identified is expanded and cropped according to the meter area detection result to obtain a processing result, and the processing result is input into the meter multimodal model together with the prompt word to obtain the pointer meter reading.

[0009] Furthermore, the target detection model includes at least a YOLO series target detection model; and the multimodal large model includes at least a CogVLM model.

[0010] Furthermore, the meter area detection dataset is obtained by horizontally boxing the pointer meter image; the meter image text dataset is obtained by providing corresponding text descriptions for the processed pointer meter image in a visual question-answering annotation format.

[0011] Furthermore, data enhancement is performed when constructing the meter area detection dataset and the meter image text dataset, including rotation, cropping, scaling, color conversion, noise addition, and special weather effect enhancement.

[0012] Furthermore, the expansion-cropping process of the pointer meter image includes:

[0013] Obtaining a horizontal frame of the marked pointer meter area; expanding four sides of the horizontal frame of the pointer meter area outward by a number of pixels;

[0014] The meter image is cropped along the boundary of the expanded frame to obtain a smaller image containing the meter area.

[0015] Furthermore, a text description is given to the processed pointer meter image to guide the multimodal large model to converge in the direction of learning the meter readings.

[0016] Furthermore, the prompt word should match the text description, and the text of the prompt word includes at least one of the keywords of the text description, the synonyms of the keywords, and the synonyms of the keywords.

[0017] Furthermore, the meter multimodal model and the meter area target detection model are distributed and locally deployed using a message queue-based service architecture, and are accelerated by a tensor acceleration engine.

[0018] Furthermore, after the pointer meter reading is identified by the method, it is determined whether the meter reading is within the warning line. If so, the reading is recorded and archived. If not, an alarm is issued while the reading is recorded and archived.

[0019] The present invention also discloses a substation pointer meter identification system based on a multi-modal large model, comprising:

[0020] Inspection equipment, used to collect images of various types of pointer meters in substations;

[0021] An expansion-cropping processing module, used for performing expansion-cropping processing on the pointer meter image;

[0022] A text description module is used to describe the pointer meter image after the expansion and cropping process, extract the prompt words, and input them into the meter multimodal model;

[0023] The meter area target detection model is used to detect the meter area of ​​the pointer meter image to be identified;

[0024] The meter multimodal model is used to identify the pointer meter reading of the image according to the pointer meter image to be identified after expansion and cropping and the prompt word.

[0025] Compared with the prior art, the present invention has the following beneficial effects: the present invention fine-tunes and improves the data set through a multimodal large model in conjunction with meter image text, and combines it with a target detection model, thereby enabling monitoring of meter readings in the substation, thereby avoiding problems such as high labor inspection costs, low efficiency, and recording errors. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 It is a flow chart of a substation pointer meter identification system based on a multi-modal large model of the present invention;

[0027] Figure 2 This is a demonstration diagram of the SF6 meter in a specific embodiment of the present invention;

[0028] Figure 3 It is a diagram of the question-answering process of the multimodal model in a specific embodiment of the present invention. DETAILED DESCRIPTION

[0029] In order to describe the technical solution in the embodiment of the present invention more clearly and completely, the technical solution of the present invention will be introduced in combination with the embodiment of the present invention and the accompanying drawings. Obviously, the described embodiment is only a part of the embodiment of the present invention, not all of the embodiments. Based on the embodiment of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0030] The present invention provides a method and system for identifying pointer meters in a substation based on a multi-modal large model, such as Figure 1 Shown is a flow chart of a specific implementation of the system of the present invention.

[0031] In this embodiment, an automatic inspection robot is selected as the platform. The main body of the robot is equipped with a mechanical arm, a pan-tilt head and an intelligent camera. It automatically takes images of pointer meters at fixed points along the inspection route in the substation through a preset path.

[0032] The automatic inspection robot collects the original images of each pointer meter in the substation based on the inspection route, and constructs the meter area detection dataset based on the original images. The original pointer meter image has a complex background. The pointer meter area in the image is annotated with a horizontal box, and the complex environment around the pointer meter is filtered out to generate an annotation file.

[0033] The expansion-cropping module obtains the original image of the pointer meter and the aforementioned annotation file, and crops the pointer meter area according to the annotated horizontal frame. In order to avoid the loss of target features in the cropped image, a certain number of pixels are expanded on the basis of the horizontal frame for cropping. Corresponding text descriptions are given to the cropped pointer meter image to construct a meter image text pair dataset, such as Figure 2 The SF6 meter is shown in the figure. The image-text pair is produced by adding question-answer annotations to the image based on visual question-answering technology. In this embodiment, the following questions are set for the pointer meter image:

[0034] 1. "What are the digital scales on the pointer meters in the picture? Arrange them in order from smallest to largest."

[0035] 2. "Which two digital scales does the pointer point to? How many small scales are there between the digital scales at both ends of the pointer? What value does each small scale represent?"

[0036] 3. "Which small scale does the pointer point to? What is the meter reading?"

[0037] In order to improve the training effect, the data set of meter image text is expanded, that is, the training images are enhanced. The means of data enhancement include rotation, cropping, flipping, scaling, color change, noise addition, and image synthesis, specifically: rotation, taking any point in the pointer meter image as the center, rotating the pointer meter image at different angles, and filling the blank part with zero value; cropping, randomly cropping the meter image without touching the meter body; scaling, changing the image size; color change, changing the color hue, saturation, and brightness values; adding noise, adding different types of noise to the training data, including Gaussian noise, salt and pepper noise, Poisson noise, speckle noise, uniform noise, multiplicative noise, and exponential noise; special weather effects enhancement, adding strong lighting, fog, rain, snow, and snow-covered special effects to the pointer meter image.

[0038] In the specific use process, one or a combination of these methods is selected to expand the original sample image into multiple images, so as to significantly increase the amount of training. During this period, the enhanced images retain the original text annotations.

[0039] A target detection model is established, and supervised learning training is performed on the target detection model through the meter area detection data set to obtain a meter area target detection model. This embodiment takes the YOLO model as an example. After the meter area detection model is verified for accuracy, it is deployed to form deployment 1. The deployment method adopts a distributed local deployment of a service architecture based on a message queue.

[0040] Specifically, the steps for verifying the accuracy of the meter area detection model are as follows:

[0041] 1. Randomly collect a certain number of pointer meter images in the substation;

[0042] 2. Send the collected pointer meter images to the above-mentioned trained target detection model for inference;

[0043] 3. Calculate the accuracy and recall of the model's inference results;

[0044] 4. The calculation method of accuracy is P = TP / (TP+FP);

[0045] 5. The calculation method of recall rate is R = TP / (TP+FN);

[0046] 6. If the accuracy rate P exceeds a certain threshold α and the recall rate is greater than a certain threshold β, the accuracy verification is passed. Otherwise, the target detection model is optimized until the accuracy rate P and recall rate R of the model are greater than or equal to the corresponding thresholds α and β;

[0047] Among them, TP is the number of pointer meters correctly detected in all image detection results, FP is the number of pointer meters incorrectly detected in all image detection results, and FN is the number of pointer meters not detected in all image detection results.

[0048] A multimodal big model is established, and the multimodal big model is trained by supervised learning through the meter image text pair dataset to obtain a meter multimodal model. A multimodal big model is a model that can process and fuse multiple different types of data modalities, and the data modalities usually include but are not limited to: text, image, audio, video. The multimodal big model integrates and understands information from these different modalities to achieve a more comprehensive and accurate understanding and analysis. The basis is to extract the features of each modality and then combine these features together using a specific fusion strategy to capture the association and complementarity between different modalities.

[0049] This embodiment takes the CogVLM model as an example. The CogVLM model not only has good performance, but its technical architecture is also more in line with the use scenario of this embodiment. The principle and use method of the CogVLM model are described in the document CogVLM: Visual Expert for Pretrained Language Models (Wang W, Lv Q, Yu W, et al. arXiv, 2023.). This model achieves deep alignment of image features and text features by adding a trainable visual expert module to each layer of the large language model, while ensuring that the performance of NLP tasks is not sacrificed while enhancing visual understanding capabilities.

[0050] In the process of supervised learning training, the CogVLM model of this embodiment needs to freeze the visual encoder and the pre-trained large language model, and only train the visual expert module to ensure that the ability of the text large model is not destroyed and that the text and visual features can be deeply jointly trained. After the accuracy of the obtained meter multimodal model is verified, it is deployed to form a deployment model 2. The deployment method adopts a distributed local deployment of a service architecture based on a message queue.

[0051] Specifically, the steps for checking the accuracy of the multimodal model of the meter are as follows:

[0052] 1. Randomly collect a certain number of pointer meter images in the substation through inspection equipment;

[0053] 2. Mark the collected images with pointer meter horizontal frames;

[0054] 3. Dilate and crop the original image according to the marked horizontal frame;

[0055] 4. Send the expanded and cropped image to the above-trained meter multimodal model for inference;

[0056] 5. Statistics of the accuracy of multimodal model reasoning results:

[0057] (1) Count the accuracy rate of answers to question 1: "What are the digital scales on the pointer meter in the picture? Arrange them in order from small to large."

[0058] (2) Count the accuracy rate of answers to question 2: “Between which two digital scales does the pointer point? How many small scales are there between the digital scales at both ends of the pointer? What value does each small scale represent?”

[0059] (3) Count the accuracy rate of answers to question 3: "Which small scale does the pointer point to? What is the meter reading?" P3;

[0060] (4) Taking the average of P1, P2, and P3, we get the model inference result accuracy mP = (P1 + P2 + P3) / 3

[0061] 6. If the accuracy rate mP of the model inference result exceeds a certain threshold value γ, the accuracy verification is passed. Otherwise, the meter multimodal model is optimized until the accuracy rate mP of the model is greater than or equal to the threshold value γ.

[0062] In the subsequent automatic inspection process, the automatic inspection robot collects the pointer meter images to be identified along the set inspection route, sends the collected pointer meter images to be identified to the deployment model 1, and performs expansion-cropping processing on the detection results. The operation steps of the expansion-cropping processing are:

[0063] 1. According to the detection results of deployment model 1, the pixel coordinates of the upper left point and lower right point of the meter detection box are obtained as p1(x1,y1) and p2(x2,y2) respectively;

[0064] 2. Based on the above two points p1 and p2, calculate the following values:

[0065] (1)x_min=x1-val if(x1-val)>0else 0

[0066] (2)y_min=y1-val if(y1-val)>0else 0

[0067] (3) x_max = x2 + val if (x2 + val) <w else w

[0068] (4)y_max=y2+val if(y2+val) <h else h

[0069] Where val is the number of expanded pixels, w and h are the width and height of the output image of the detection result of deployment model 1 respectively;

[0070] 3. Cut the original image ori_image before inputting model 1 to obtain the cropped image cut_image:

[0071] cut_image=ori_image[y_min:y_max,x_min:x_max]

[0072] cut_image is the result after expansion and cropping;

[0073] The cropped image and the prompt word text are input into the deployment model 2 together to obtain the meter reading result. Among them, the text in the pointer meter image text pair is matched with the prompt word text; and the prompt word text is provided with at least the question keywords annotated in the text of the image text pair, or the synonyms of the keywords, or the synonyms of the keywords. This design enables the meter multimodal model to obtain the reading result of the pointer meter in the form of question and answer. Figure 3 The following is a question and answer diagram of the multimodal model in this embodiment. The prompt words are set according to the questions annotated in the text:

[0074] 1. "What are the digital scales on the pointer meters in the picture? Arrange them in order from smallest to largest."

[0075] 2. "Which two digital scales does the pointer point to? How many small scales are there between the digital scales at both ends of the pointer? What value does each small scale represent?"

[0076] 3. "Which small scale does the pointer point to? What is the meter reading?"

[0077] The answers were:

[0078] 1. "The pointer meter in the picture has these digital scales. They are in ascending order: -0.1, 0, 0.2, 0.4, 0.6, 0.8, 0.9"

[0079] 2. "The pointer points to the digital scale between 0.4 and 0.6. There are 10 small scales between 0.4 and 0.6. The value of each small scale represents (0.6-0.4) / 10=0.02"

[0080] 3. "The pointer points to the sixth small scale between 0.4 and 0.6. Each small scale represents a value of 0.02, so the meter reading is 0.4+6*0.02=5.2"

[0081] After the system obtains the reading result, it determines whether the reading result is within the warning line. If so, the reading is recorded and archived. If not, an alarm is issued while recording and archiving the reading. The warning line is a threshold. If the meter reading exceeds this threshold, it is considered to be an abnormal situation and an alarm is issued. Figure 2 The pointer meter reading shown is 0.52. If the threshold is 0.6 or greater, it is considered that it does not exceed the warning line. If the threshold is 0.5, it is considered that the meter reading has exceeded the warning line and an alarm needs to be issued.

[0082] What is described above is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A method for identifying pointer meters in a substation based on a multi-modal large model, characterized in that the steps include: The inspection equipment collects original images of various types of pointer meters in the substation and constructs a meter area detection data set based on the original images; Establishing a target detection model, and performing supervised learning training on the target detection model using the meter area detection data set to obtain a meter area target detection model; Perform expansion-cropping processing on the original image of the pointer meter, give a text description to the pointer meter image after the expansion-cropping processing, extract the prompt words, and construct a meter image-text pair data set; establish a multimodal large model, and perform supervised learning training on the multimodal large model through the meter image-text pair data set to obtain a meter multimodal model; The inspection equipment collects the original image of the pointer meter to be identified, and inputs the original image of the pointer meter to be identified into the meter area target detection model to obtain the meter area detection result; the pointer meter image to be identified is expanded and cropped according to the meter area detection result to obtain the processing result, and the processing result is input into the meter multimodal model together with the prompt word to obtain the pointer meter reading.

2. The method for identifying a pointer meter in a substation according to claim 1, characterized in that: The target detection model includes at least a YOLO series target detection model; and the multimodal large model includes at least a CogVLM model.

3. The method for identifying a pointer meter in a substation according to claim 1, characterized in that: The meter area detection dataset is obtained by annotating the pointer meter image with a horizontal frame; the meter image text dataset is obtained by annotating the processed pointer meter image with a corresponding text description in a visual question-answering annotation form.

4. The method for identifying a pointer meter in a substation according to claim 1, characterized in that: When constructing the meter area detection dataset and the meter image text pair dataset, data enhancement is performed, including rotation, cropping, scaling, color conversion, noise addition, and special weather effect enhancement.

5. The method for identifying a pointer meter in a substation according to claim 1, characterized in that: The expansion-cropping process of the pointer meter image includes: Obtaining a horizontal frame of the marked pointer meter area; expanding four sides of the horizontal frame of the pointer meter area outward by a number of pixels; The meter image is cropped along the boundary of the expanded frame to obtain a smaller image containing the meter area.

6. The method for identifying a pointer meter in a substation according to claim 1, characterized in that: The processed pointer meter image is described in words to guide the multimodal large model to converge in the direction of learning meter readings.

7. The method for identifying a pointer meter in a substation according to claim 1, characterized in that: The prompt word should match the text description, and the text of the prompt word includes at least one of the keywords of the text description, the synonyms of the keywords, and the synonyms of the keywords.

8. The method for identifying a pointer meter in a substation according to claim 1, characterized in that: The meter multimodal model and the meter area target detection model are locally deployed in a distributed manner using a message queue-based service architecture, and are accelerated by a tensor acceleration engine.

9. The method for identifying a pointer meter in a substation according to claim 1, characterized in that: After the pointer meter reading is identified by the method, it is determined whether the meter reading is within the warning line. If so, the reading is recorded and archived. If not, an alarm is issued while the reading is recorded and archived.

10. A substation pointer meter identification system based on a multi-modal large model, characterized in that: include: Inspection equipment, used to collect original images of various types of pointer meters in substations; An expansion-cropping processing module is used to perform expansion-cropping processing on the original image of the pointer meter; A text description module is used to describe the pointer meter image after the expansion and cropping process, extract the prompt words, and input them into the meter multimodal model; The meter area target detection model is used to detect the meter area of ​​the pointer meter image to be identified; The meter multimodal model is used to identify the pointer meter reading of the image according to the pointer meter image to be identified after expansion and cropping and the prompt word.

Images