A substation robot patrol inspection identification method and system

Abstract

The application discloses a substation robot inspection identification method and system, comprising: a substation equipment real-time video stream is collected by an inspection robot camera, images are intercepted at an interval of 1 frame per several seconds, and through a definition detection filtering fuzzy invalid frame, a to-be-identified image is obtained; the to-be-identified image is input into a multimodal large model, equipment type classification is completed, and the equipment types include a pointer instrument, a multi-row digital instrument, a power cabinet and an infrared image; a differentiated processing procedure is executed according to different equipment types, and reading, state or characteristic information of each equipment is output; a standardization JSON format record is generated by integrating a frame time stamp, an equipment type identifier, an identification result and an identification state, and is stored. Through the multimodal large model, accurate classification of various equipment types is realized, differentiated processing procedures are designed for different equipment, equipment information such as a pointer instrument, a multi-row digital instrument, a power cabinet and an infrared image can be efficiently identified, and the application is suitable for a complex equipment environment of a substation.

Description

Technical Field

[0001] This invention relates to the field of intelligent inspection technology for power systems, specifically to a substation robot inspection and identification method and system. Background Technology

[0002] As the core hub of the power system, the operating status of substation equipment directly affects the security and stability of power supply. Traditional substation equipment inspection mainly relies on manual labor. However, with the development of robotics and artificial intelligence technologies, robot inspection of substations has become the mainstream trend replacing manual inspection. Existing robot inspection and identification methods for substations are mostly designed for single types of equipment, resulting in the following shortcomings: First, poor adaptability to multiple devices, making it difficult to actively distinguish between different types of equipment such as pointer instruments, digital instruments, power cabinets, and infrared imaging devices, and to use specific identification methods for each. Second, identification accuracy is greatly affected by the environment; changes in lighting at the substation site, equipment contamination, and image tilt can easily lead to identification errors. Third, low technological integration; most methods rely on only a single machine learning algorithm or model, failing to fully leverage the synergistic advantages of different technologies.

[0003] Multimodal large-scale models possess powerful semantic understanding and cross-modal information processing capabilities, while machine learning algorithms have proven effective in image preprocessing and object detection. Integrating these two approaches for substation robot inspection and recognition can effectively improve the accuracy and adaptability of multi-device identification. Therefore, there is an urgent need for a substation robot inspection and recognition method based on the fusion of multimodal large-scale models and machine learning to address the aforementioned problems in existing technologies. Summary of the Invention

[0004] The purpose of this invention is to provide a substation robot inspection and identification method and system to solve the problems mentioned in the background art.

[0005] To achieve the above objectives, the present invention provides the following technical solution: A substation robot inspection and identification method includes: Step 1: The inspection robot's camera collects real-time video streams of the substation equipment, captures images at intervals of one frame every few seconds, filters out blurry and invalid frames through clarity detection, and obtains the image to be identified. Step 2: Input the image to be identified into the multimodal large model, and complete the equipment type classification through the multimodal large model. The equipment types include pointer instruments, multi-line digital instruments, power cabinets, and infrared images. Step 3: Perform differentiated processing according to different device types, and output the readings, status or characteristic information of each device; Step 4: Integrate the frame timestamp, device type identifier, recognition result, and recognition status to generate a standardized JSON format record and store it.

[0006] Furthermore, in step 1, the sharpness detection adopts the Laplacian variance method, with a preset sharpness threshold ≥50. Images below the sharpness threshold are judged as blurry and invalid frames.

[0007] Further, step 3 includes: For pointer meters and multi-line digital meters: execute the fusion recognition process of "machine learning preprocessing + multimodal large model interpretation"; For power cabinets: Perform YOLO target detection and morphological feature analysis; For infrared camera images: execute the "OCR recognition of maximum / minimum temperature values ​​+ temperature threshold detection" process.

[0008] Furthermore, when the device type in step 3 is a pointer instrument, the corresponding fusion identification includes: S100 performs mean-shift filtering noise reduction on the pointer instrument image, and then further reduces noise interference through Gaussian filtering to enhance the contrast between image edges and background; S101, extract the dial outline using the Canny edge detection algorithm, and calculate the geometric feature parameters based on the Hough circle transform. The geometric feature parameters include the dial center coordinates, radius, tilt angle, and coordinates of the circumscribed rectangle vertices. S102, construct a perspective transformation matrix based on the geometric feature parameters, perform perspective correction on the image, and generate a front-view dial image; S103, perform adaptive binarization processing on the front-view dial image, extract all contours in the image, separate candidate contours of the scale lines and candidate contours of the pointer through contour feature filtering, remove abnormal contours, and generate a mask of effective scale line contours and candidate contours of the pointer. S104, perform straight line fitting on the effective scale line contour, and determine the precise geometric center of the dial by statistically analyzing the intersection points of multiple sets of scale line fitting lines; S105, Perform line detection on the pointer mask image, filter out valid pointer line segments, and locate the far end of the pointer; S106: Input the preprocessed front view image and extracted feature data into the multimodal large model, match the pointer angle and scale value through the semantic understanding of the large model, and output the pointer instrument reading.

[0009] Furthermore, the perspective correction method in S102 includes: S1020, determine the size of the target front view image according to the radius described in S101, and set the coordinates of the four vertices of the rectangle corresponding to the target front view image; S1021, Based on the coordinates of the four vertices of the bounding rectangle of the original dial image and the coordinates of the four vertices of the rectangle of the target front view image, a perspective transformation matrix is ​​generated by a perspective transformation matrix calculation algorithm. S1022, Input the original dial image into the perspective transformation matrix to calculate the corresponding coordinates of each pixel in the original tilted dial image of the target front view image, and perform perspective transformation using a bilinear interpolation algorithm to eliminate dial tilt distortion and generate a distortion-free front view dial image. The implementation of contour feature filtering and abnormal contour removal in S103 includes: S1030: Based on the minimum aspect ratio of the bounding rectangle of the contour, filter slender contours with an aspect ratio ≥ 4 as candidate contours for scale lines, and filter candidate contours for pointers based on the contour size. S1031, calculate the interquartile range of the candidate scale line contour area using the box plot method, and remove abnormal contours whose area exceeds the range of [Q1-1.5×IQR, Q3+2.5×IQR] to obtain the effective scale line contours. Here, Q1 is the lower quartile of the candidate scale line contour area, Q3 is the upper quartile of the candidate scale line contour area, and IQR is the interquartile range, calculated by the formula IQR=Q3-Q1. S1032, generate masks for the effective scale line contour and the pointer candidate contour respectively; The method for determining the precise geometric center of the dial in S104 includes: S1040, perform minimum bounding rectangle fitting and least squares-based line fitting on the profile of each valid scale line to obtain the slope and intercept of the scale line. S1041, randomly sample multiple sets of scale lines to fit straight lines, and calculate the coordinates of the intersection points of each pair of straight lines; S1042, take the mean of the x-coordinates and y-coordinates of all valid intersection points as the precise geometric center of the dial; The implementation of valid pointer line segment detection and pointer distal end point positioning in S105 includes: S1050 performs a probabilistic Hough linear transform on the pointer mask image; S1051, select the longest line segment in the probability Hough linear transform detection results as the valid pointer; S1052, calculate the Euclidean distance from the two endpoints of the straight line segment to the precise geometric center of the dial, and take the endpoint with the greater distance as the farthest endpoint of the pointer.

[0010] Furthermore, when the device type in step 3 is a multi-row digital meter, the corresponding fusion identification includes: S200 performs grayscale and adaptive threshold binarization processing on multi-line digital instrument images to improve the contrast between the numbers and the background. S201, a multi-line digital image is divided into independent single-line regions using a horizontal projection method. Methods for splitting multi-line digital images include: S2010, calculate the horizontal projection histogram of the image, count the number of foreground pixels in each row, and identify the coordinates of the dividing line between the numbered rows and the blank rows. S2011, the original image is cropped vertically based on the boundary row coordinates to obtain multiple independent image regions containing only single-row numbers; S202 inputs the segmented region image into the multimodal large model, identifies and integrates the numbers line by line, and outputs the complete readings of the multi-line digital instrument.

[0011] Furthermore, when the equipment type in step 3 is a power cabinet, the corresponding processing includes: S300 uses a pre-trained YOLO model. The training dataset of the YOLO model already includes target labels for switches, indicator lights of different colors and statuses, and embedded meters on the substation power cabinet panel. It performs target detection on the power cabinet panel image, locates the bounding box, category, and detection confidence of each target. S301, for indicator light targets, directly extracts pre-labeled color categories and on / off status information from the YOLO model detection results, and determines the operating status of the power cabinet according to preset rules; S302, for the switch target, directly extract the pre-labeled on / off state of the switch from the YOLO model detection results; S303, targeting embedded electricity meters, first performs grayscale and adaptive threshold binarization on the image to improve the contrast between the numbers and the background. Then, it splits the image into independent single-line regions using the horizontal projection method, inputs a multimodal large model to identify the numbers line by line and integrates them to complete the reading extraction. S304, based on the detection results of each target, outputs a result image containing YOLO category detection boxes, confidence scores, power cabinet operating status, and embedded meter recognition results.

[0012] Furthermore, when the device type in step 3 is an infrared camera image, the corresponding processing includes: The S400 uses an OCR algorithm to identify the maximum and minimum temperature values ​​marked on infrared image frames, focuses on the top 5%-30% of the marked area of ​​the image, and extracts the effective temperature values ​​through regular expression matching. S401 converts infrared images into grayscale images to reduce background interference; S402: Based on the temperature extreme values ​​identified by OCR, a linear mapping relationship is constructed, and the grayscale image pixel values ​​are normalized and mapped to the actual temperature values ​​to generate an infrared image temperature distribution map. S403, based on a preset temperature threshold, detects hotspot areas exceeding the threshold in the temperature distribution map, extracts statistical information such as hotspot coordinates, hotspot area percentage, average hotspot temperature, and highest hotspot temperature, and outputs an infrared temperature detection result image containing hotspot analysis information and overheated area detection.

[0013] Furthermore, the standardized JSON format record in step 4 includes timestamp, device_type, result, and status fields, wherein: ① The result field of pointer meters / multi-line digital meters is the reading + unit; ②The result field of the power cabinet is the path of the cabinet panel result image recognized by the YOLO model; ③ The result field of the infrared image is the path of the infrared temperature detection result image.

[0014] The present invention also provides a substation robot inspection and identification system for implementing the substation robot inspection and identification method described above, characterized in that it includes: An image acquisition and preprocessing module is used to implement step 1 of the substation robot inspection and identification method. A multimodal equipment classification module is used to implement step 2 of the substation robot inspection and identification method. A differential identification processing module, which is used to implement step 3 of the substation robot inspection and identification method; and The data integration and standardization storage module is used to implement step 4 of the substation robot inspection and identification method.

[0015] Compared with the prior art, the beneficial effects of the present invention are: 1. Compared with existing technologies that struggle to actively distinguish different equipment types using specific identification methods, this invention achieves accurate classification of various equipment types through a multimodal large model. It designs differentiated processing flows for different equipment, which can efficiently identify equipment information such as pointer instruments, multi-line digital instruments, power cabinets, and infrared images, adapting to the complex equipment environment of substations.

[0016] 2. Compared with existing pointer instruments and multi-line digital instruments whose recognition accuracy is greatly affected by the environment, this invention optimizes image quality through machine learning preprocessing units (such as noise reduction enhancement, perspective correction, grayscale binarization, etc.) in each processing sub-module. Combined with the semantic understanding advantages of multimodal large models, it significantly improves recognition accuracy and effectively reduces the impact of environmental interference on recognition results.

[0017] 3. Compared with the existing technologies, which have low integration and rely on only a single algorithm or model, this invention constructs a fusion architecture of "machine learning preprocessing + multimodal large model interpretation", which gives full play to the advantages of different technologies in image preprocessing, object detection, semantic understanding, etc., to achieve synergistic effect and improve overall recognition performance. Attached Figure Description

[0018] Figure 1 This is an overall flowchart of a substation robot inspection and identification method according to the present invention; Figure 2 This is a detailed flowchart of the pointer instrument fusion recognition process in a substation robot inspection and recognition method of the present invention; Figure 3 This is a detailed flowchart of the multi-line digital instrument fusion recognition process in the substation robot inspection and recognition method of the present invention; Figure 4 This is a detailed flowchart of the power cabinet processing flow in the substation robot inspection and identification method of the present invention; Figure 5 This is a detailed flowchart of the infrared image processing process in a substation robot inspection and identification method according to the present invention.

[0019] Figure 6 This is a schematic diagram of the structure of a substation robot inspection and identification system according to the present invention. Detailed Implementation

[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] Please see Figure 1 A substation robot inspection and identification method, comprising: Step 1: The inspection robot's camera captures real-time video streams of the substation equipment, extracting images at 2-second frame intervals. Blurry and invalid frames are filtered out using a sharpness detection system to obtain the image to be identified. In other embodiments, images can be extracted at 1-second frame intervals, or at 3-second or 4-second intervals.

[0022] The sharpness detection employs the Laplacian variance method. First, the cropped color image is converted to grayscale to obtain a grayscale image. Then, the edge response values ​​of the image are calculated using a two-dimensional Laplacian operator. Finally, the variance of the response values ​​is calculated as the sharpness evaluation index. The sharpness calculation formula is as follows: (1) (2) In equations (1) and (2), σ 2is the Laplacian variance, i.e., the image sharpness value; the larger the value, the sharper the image; W is the pixel width of the grayscale image; H is the pixel height of the grayscale image; i and j are the horizontal and vertical coordinate indices of the pixels in the grayscale image; L(i,j) is the edge response value of pixel (i,j) after convolution with the Laplacian operator. This is the average edge response value of all pixels. A preset sharpness threshold of ≥50 is used; images below this threshold are considered blurry and invalid frames.

[0023] Step 2: Input the image to be identified into the locally deployed multimodal large model, and complete the device type classification through the multimodal large model. The device types include pointer instruments, multi-line digital instruments, power cabinets, and infrared images.

[0024] Step 3: Perform differentiated processing according to different device types, and output the readings, status or characteristic information of each device.

[0025] For pointer meters and multi-line digital meters: execute the "machine learning preprocessing + multimodal large model interpretation" fusion recognition process.

[0026] For power cabinets: Perform the YOLO target detection and morphological feature analysis process.

[0027] For infrared camera images: execute the "OCR recognition of maximum / minimum temperature values ​​+ temperature threshold detection" process.

[0028] like Figure 2 As shown, when step 2 determines the device type to be a pointer meter, the "machine learning preprocessing + multimodal large model interpretation" fusion recognition process is executed, including: S100 performs mean-shift filtering noise reduction on the pointer instrument image, and then further reduces noise interference through Gaussian filtering to enhance the contrast between image edges and background.

[0029] S101, the dial outline is extracted using the Canny edge detection algorithm, and geometric feature parameters are calculated based on the Hough circle transform. The geometric feature parameters include the dial center coordinates, radius, tilt angle, and coordinates of the circumscribed rectangle vertices.

[0030] S102, construct a perspective transformation matrix based on the geometric feature parameters, perform perspective correction on the image, and generate a front-view dial image. The perspective correction method includes: S1020, determine the size of the target front view image according to the radius described in S101, and set the coordinates of the four vertices of the rectangle corresponding to the target front view image.

[0031] S1021, Based on the coordinates of the four vertices of the bounding rectangle of the original dial image and the coordinates of the four vertices of the rectangle of the target front view image, a perspective transformation matrix is ​​generated using a perspective transformation matrix calculation algorithm. The perspective transformation matrix is ​​a 3×3 matrix, in the form of: (3) This matrix satisfies the pixel mapping relationship: (4) In equations (3) and (4), H is a 3×3 perspective transformation matrix, and the matrix element h 11 h 12 h 13 h 21 h 22 h 23 h 31 h 32 The unknowns to be solved are solved by solving the equations of the original and target vertex coordinates according to equation (4); (x,y) are the homogeneous coordinates of the pixel in the original dial image; (x',y') are the homogeneous coordinates of the corresponding pixel in the front view image of the target.

[0032] S1022, Input the original dial image into the perspective transformation matrix to calculate the corresponding coordinates of each pixel in the original tilted dial image. Based on the matrix mapping result, perform perspective transformation using a bilinear interpolation algorithm to eliminate dial tilt distortion and generate a distortion-free front-view dial image. The pixel grayscale value calculation formula for the bilinear interpolation algorithm is: (5) Where u and v are intermediate quantities, u = x' - x o v=y'-y o In equation (5), I(x',y') is the gray value of the interpolation point (x',y') in the front view image of the target; (x0,y0), (x1,y0), (x0,y1), and (x1,y1) are the coordinates of the four neighboring pixels surrounding the interpolation point after perspective matrix mapping, and x1=x o +1, y1=y o +1; I(x0,y0), I(x1,y0), I(x0,y1), I(x1,y1) are the gray values ​​of four neighboring pixels in the original image; u and v are the horizontal and vertical offsets of the interpolation point relative to the neighboring pixel (x0,y0), with values ​​ranging from [0,1].

[0033] S103 performs adaptive binarization on the front-view dial image, extracts all contours in the image, separates candidate contours for the tick marks and pointers through contour feature filtering, removes abnormal contours, and generates a mask for the valid tick mark contours and pointer candidate contours. The implementation of contour feature filtering and abnormal contour removal includes: S1030: Based on the minimum aspect ratio of the bounding rectangle of the contour, select slender contours with an aspect ratio ≥ 4 as candidate contours for scale lines, and select candidate contours for pointers based on the contour size.

[0034] S1031. The interquartile range (IQR) of the candidate scale line contour area is calculated using the box plot method. Abnormal contours with areas exceeding the range of [Q1-1.5×IQR, Q3+2.5×IQR] are removed to obtain the effective scale line contours. Here, Q1 is the lower quartile of the candidate scale line contour area (corresponding to the 25th percentile of the area data), Q3 is the upper quartile of the candidate scale line contour area (corresponding to the 75th percentile of the area data), and IQR is the interquartile range, calculated by the formula IQR=Q3-Q1, which is used to characterize the dispersion of the candidate scale line contour area.

[0035] S1032, generate masks for the effective scale line contour and the pointer candidate contour, respectively.

[0036] S104, perform straight line fitting on the effective scale line contours, and determine the precise geometric center of the dial by statistically analyzing the intersection points of multiple sets of fitted scale line lines. The methods for determining the precise geometric center of the dial include: S1040, perform minimum bounding rectangle fitting and least squares-based line fitting on the profile of each valid scale line to obtain the slope and intercept of the scale line. The solution formula for the least squares-based line fitting is: (6) (7) In equations (6) and (7), k is the slope of the fitted line of the scale line, b is the intercept of the fitted line, n is the number of pixels on the scale line outline, (x m ,y m ) represents the coordinates of the m-th pixel on the scale line outline.

[0037] S1041, randomly sample multiple sets of scale lines to fit straight lines, and calculate the coordinates of the intersection points of each pair of straight lines.

[0038] S1042, take the mean of the x-coordinates and y-coordinates of all valid intersection points as the precise geometric center of the dial.

[0039] S105, perform line detection on the pointer mask image, filter out valid pointer line segments, and locate the pointer's distal end. The implementation of valid pointer line segment detection and pointer distal end location includes: S1050 performs a probabilistic Hough linear transform on the pointer mask image.

[0040] S1051, select the longest line segment from the probabilistic Hough linear transform detection results as the valid pointer.

[0041] S1052, Calculate the Euclidean distance from the two endpoints of the straight line segment to the precise geometric center of the dial. The formula for calculating the Euclidean distance is: (8) In equation (8), d is the Euclidean distance from the pixel to the precise geometric center of the dial; (x p ,y p (x) represents the coordinates of any pixel point on the pointer line segment; c ,y c ) represents the pixel coordinates of the precise geometric center of the dial, with the more distant endpoints used as the pointer's farthest endpoints.

[0042] S106: Input the preprocessed front view image and extracted feature data into the multimodal large model, match the pointer angle and scale value through the semantic understanding of the large model, and output the pointer instrument reading.

[0043] like Figure 3 As shown, when step 2 determines the device type to be a multi-row digital meter, the "machine learning preprocessing + multimodal large model interpretation" fusion recognition process is executed, including: The S200 performs grayscale and adaptive threshold binarization processing on multi-line digital instrument images to improve the contrast between the numbers and the background.

[0044] S201, a multi-line digital image is divided into independent single-line regions using a horizontal projection method. Methods for splitting multi-line digital images include: S2010: Calculate the horizontal projection histogram of the image, count the number of foreground pixels in each row, and identify the coordinates of the dividing line between the numbered rows and the blank lines.

[0045] S2011, the original image is vertically cropped based on the boundary row coordinates to obtain multiple independent image regions containing only single-row numbers.

[0046] S202 inputs the segmented region image into the multimodal large model, identifies and integrates the numbers line by line, and outputs the complete readings of the multi-line digital instrument.

[0047] like Figure 4 As shown, when step 2 determines the equipment type to be a power cabinet, the YOLO target detection and analysis process is executed, and the corresponding processing includes: S300 uses a pre-trained YOLO model, which can be YOLO11. The training dataset of the YOLO model already includes target labels such as switches, indicator lights of different colors (red / green / white) and statuses, and embedded meters on the substation power cabinet panel. It performs target detection on the power cabinet panel image, and locates the bounding box, category and detection confidence of each target.

[0048] S301, for indicator light targets, directly extracts pre-labeled color categories and on / off status information from the YOLO model detection results, and determines the operating status of the power cabinet according to preset rules.

[0049] S302, for the switch target, directly extracts the pre-labeled on / off state of the switch from the YOLO model detection results.

[0050] S303, targeting embedded electricity meters, first performs grayscale and adaptive threshold binarization on the image to improve the contrast between the numbers and the background. Then, it splits the image into independent single-line regions using the horizontal projection method, inputs a multimodal large model to identify the numbers line by line and integrates them to complete the reading extraction.

[0051] S304, based on the detection results of each target, outputs a result image containing YOLO category detection boxes, confidence scores, power cabinet operating status, and embedded meter recognition results.

[0052] like Figure 5 As shown, when step 2 determines the device type to be an infrared camera image, the "OCR recognition of extreme temperature values ​​+ temperature threshold detection" process is executed, including: The S400 uses an OCR algorithm to identify the maximum and minimum temperature values ​​marked on the infrared image frame, focuses on the top 5%-30% (preferably 10%) of the marked area of ​​the image, and extracts the effective temperature values ​​through regular expression matching.

[0053] S401 converts infrared images into grayscale images to reduce background interference.

[0054] S402, a linear mapping relationship is constructed based on the temperature extreme values ​​identified by OCR. The normalized pixel values ​​of the grayscale image are mapped to the actual temperature values. The formula for calculating the linear temperature mapping is: (9) In equation (9), T(G) is the actual temperature value corresponding to the pixel with a gray value of G; G is the gray value of the pixel in the infrared grayscale image, with a value range of [0, 255]; G min G represents the minimum grayscale value in the infrared grayscale image. max T represents the maximum grayscale value in the infrared grayscale image. min The minimum temperature of the infrared image identified by OCR; T max The maximum temperature of the infrared image identified by OCR is calculated using Equation (9) to generate an infrared image temperature distribution map. S403, based on a preset temperature threshold, detects hotspot areas exceeding the threshold in the temperature distribution map, extracts statistical information such as hotspot coordinates, hotspot area percentage, average hotspot temperature, and highest hotspot temperature, and outputs an infrared temperature detection result image containing hotspot analysis information and overheated area detection.

[0055] Step 4: Integrate the frame timestamp, device type identifier, recognition result, and recognition status to generate a standardized JSON format record and store it.

[0056] Standardized JSON format records include timestamp, device_type, result, and status fields, among which: ① The result field of pointer meters / multi-line digital meters is the reading + unit; ②The result field of the power cabinet is the path of the cabinet panel result image recognized by the YOLO model; ③ The result field of the infrared image is the path of the infrared temperature detection result image.

[0057] Please see Figure 6 A substation robot inspection and identification system is provided to implement the substation robot inspection and identification method described above. The system includes an image acquisition and preprocessing module, a multimodal equipment classification module, a differential identification processing module, and a data integration and standardized storage module. All four modules are computer programs.

[0058] The image acquisition and preprocessing module is used to implement step 1 of the substation robot inspection and identification method.

[0059] The multimodal equipment classification module is used to implement step 2 of the above-mentioned substation robot inspection and identification method.

[0060] The differential recognition processing module is used to implement step 3 of the above-mentioned substation robot inspection and recognition method. The differential recognition processing module includes a pointer instrument fusion recognition submodule, a multi-line digital instrument fusion recognition submodule, a power cabinet panel detection submodule, and an infrared image temperature analysis submodule.

[0061] The pointer instrument fusion recognition submodule is used to implement the fusion recognition process of the pointer instruments in step 3 of the above substation robot inspection and recognition method.

[0062] The multi-line digital instrument fusion recognition submodule is used to implement the fusion recognition process corresponding to the multi-line digital instruments in step 3 of the above substation robot inspection and recognition method.

[0063] The data integration and standardized storage module is used to implement the power cabinet processing in step 3 of the above substation robot inspection and identification method.

[0064] The infrared image temperature analysis submodule is used to implement the infrared camera image processing process in step 3 of the above-mentioned substation robot inspection and identification method.

[0065] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for robot inspection and identification in substations, characterized in that, include: Step 1: The inspection robot's camera collects real-time video streams of the substation equipment, captures images at intervals of one frame every few seconds, filters out blurry and invalid frames through clarity detection, and obtains the image to be identified. Step 2: Input the image to be identified into the multimodal large model, and complete the equipment type classification through the multimodal large model. The equipment types include pointer instruments, multi-line digital instruments, power cabinets, and infrared images. Step 3: Perform differentiated processing according to different device types, and output the readings, status or characteristic information of each device; Step 4: Integrate the frame timestamp, device type identifier, recognition result, and recognition status to generate a standardized JSON format record and store it.

2. The substation robot inspection and identification method according to claim 1, characterized in that, In step 1, the sharpness detection uses the Laplacian variance method, with a preset sharpness threshold of ≥50. Images below the sharpness threshold are judged as blurry and invalid frames.

3. The substation robot inspection and identification method according to claim 1, characterized in that, Step 3 includes: For pointer meters and multi-line digital meters: execute the "machine learning preprocessing + multimodal large model interpretation" fusion recognition process; For power cabinets: Perform YOLO target detection and morphological feature analysis; For infrared camera images: execute the "OCR recognition of maximum / minimum temperature values ​​+ temperature threshold detection" process.

4. The substation robot inspection and identification method according to claim 1, characterized in that, When the device type in step 3 is a pointer meter, the corresponding fusion identification includes: S100 performs mean-shift filtering noise reduction on the pointer instrument image, and then further reduces noise interference through Gaussian filtering to enhance the contrast between image edges and background; S101, extract the dial outline using the Canny edge detection algorithm, and calculate the geometric feature parameters based on the Hough circle transform. The geometric feature parameters include the dial center coordinates, radius, tilt angle, and coordinates of the circumscribed rectangle vertices. S102, construct a perspective transformation matrix based on the geometric feature parameters, perform perspective correction on the image, and generate a front-view dial image; S103, perform adaptive binarization processing on the front-view dial image, extract all contours in the image, separate candidate contours of the scale lines and candidate contours of the pointer through contour feature filtering, remove abnormal contours, and generate a mask of effective scale line contours and candidate contours of the pointer. S104, perform straight line fitting on the effective scale line contour, and determine the precise geometric center of the dial by statistically analyzing the intersection points of multiple sets of scale line fitting lines; S105, Perform line detection on the pointer mask image, filter out valid pointer line segments, and locate the far end of the pointer; S106: Input the preprocessed front view image and extracted feature data into the multimodal large model, match the pointer angle and scale value through the semantic understanding of the large model, and output the pointer instrument reading.

5. The substation robot inspection and identification method according to claim 4, characterized in that, The method for fluoroscopic correction in S102 includes: S1020, determine the size of the target front view image according to the radius described in S101, and set the coordinates of the four vertices of the rectangle corresponding to the target front view image; S1021, Based on the coordinates of the four vertices of the bounding rectangle of the original dial image and the coordinates of the four vertices of the rectangle of the target front view image, a perspective transformation matrix is ​​generated by a perspective transformation matrix calculation algorithm. S1022, Input the original dial image into the perspective transformation matrix to calculate the corresponding coordinates of each pixel in the original tilted dial image of the target front view image, and perform perspective transformation using a bilinear interpolation algorithm to eliminate dial tilt distortion and generate a distortion-free front view dial image. The implementation of contour feature filtering and abnormal contour removal in S103 includes: S1030: Based on the minimum aspect ratio of the bounding rectangle of the contour, filter slender contours with an aspect ratio ≥ 4 as candidate contours for scale lines, and filter candidate contours for pointers based on the contour size. S1031, calculate the interquartile range of the candidate scale line contour area using the box plot method, and remove abnormal contours whose area exceeds the range of [Q1-1.5×IQR, Q3+2.5×IQR] to obtain the effective scale line contours. Here, Q1 is the lower quartile of the candidate scale line contour area, Q3 is the upper quartile of the candidate scale line contour area, and IQR is the interquartile range, calculated by the formula IQR=Q3-Q1. S1032, generate masks for the effective scale line contour and the pointer candidate contour respectively; The method for determining the precise geometric center of the dial in S104 includes: S1040, perform minimum bounding rectangle fitting and least squares-based line fitting on the profile of each valid scale line to obtain the slope and intercept of the scale line. S1041, randomly sample multiple sets of scale lines to fit straight lines, and calculate the coordinates of the intersection points of each pair of straight lines; S1042, take the mean of the x-coordinates and y-coordinates of all valid intersection points as the precise geometric center of the dial; The implementation of valid pointer line segment detection and pointer distal end point positioning in S105 includes: S1050 performs a probabilistic Hough linear transform on the pointer mask image; S1051, select the longest line segment in the probability Hough linear transform detection results as the valid pointer; S1052, calculate the Euclidean distance from the two endpoints of the straight line segment to the precise geometric center of the dial, and take the endpoint with the greater distance as the farthest endpoint of the pointer.

6. The substation robot inspection and identification method according to claim 1, characterized in that, When the device type in step 3 is a multi-row digital meter, the corresponding fusion identification includes: S200 performs grayscale and adaptive threshold binarization processing on multi-line digital instrument images to improve the contrast between the numbers and the background. S201, a multi-line digital image is divided into independent single-line regions using a horizontal projection method. Methods for splitting multi-line digital images include: S2010, calculate the horizontal projection histogram of the image, count the number of foreground pixels in each row, and identify the coordinates of the dividing line between the numbered rows and the blank rows. S2011, the original image is cropped vertically based on the boundary row coordinates to obtain multiple independent image regions containing only single-row numbers; S202 inputs the segmented region image into the multimodal large model, identifies and integrates the numbers line by line, and outputs the complete readings of the multi-line digital instrument.

7. The substation robot inspection and identification method according to claim 1, characterized in that, When the equipment type in step 3 is a power cabinet, the corresponding processing includes: S300 uses a pre-trained YOLO model. The training dataset of the YOLO model already includes target labels for switches, indicator lights of different colors and statuses, and embedded meters on the substation power cabinet panel. It performs target detection on the power cabinet panel image, locates the bounding box, category, and detection confidence of each target. S301, for indicator light targets, directly extracts pre-labeled color categories and on / off status information from the YOLO model detection results, and determines the operating status of the power cabinet according to preset rules; S302, for the switch target, directly extract the pre-labeled on / off state of the switch from the YOLO model detection results; S303, targeting embedded electricity meters, first performs grayscale and adaptive threshold binarization on the image to improve the contrast between the numbers and the background. Then, it splits the image into independent single-line regions using the horizontal projection method, inputs a multimodal large model to identify the numbers line by line and integrates them to complete the reading extraction. S304, based on the detection results of each target, outputs a result image containing YOLO category detection boxes, confidence scores, power cabinet operating status, and embedded meter recognition results.

8. The substation robot inspection and identification method according to claim 1, characterized in that, When the device type in step 3 is an infrared camera image, the corresponding processing includes: The S400 uses an OCR algorithm to identify the maximum and minimum temperature values ​​marked on infrared image frames, focuses on the top 5%-30% of the marked area of ​​the image, and extracts the effective temperature values ​​through regular expression matching. S401 converts infrared images into grayscale images to reduce background interference; S402: Based on the temperature extreme values ​​identified by OCR, a linear mapping relationship is constructed, and the grayscale image pixel values ​​are normalized and mapped to the actual temperature values ​​to generate an infrared image temperature distribution map. S403, based on a preset temperature threshold, detects hotspot areas exceeding the threshold in the temperature distribution map, extracts statistical information such as hotspot coordinates, hotspot area percentage, average hotspot temperature, and highest hotspot temperature, and outputs an infrared temperature detection result image containing hotspot analysis information and overheated area detection.

9. A substation robot inspection and identification method according to claim 1, characterized in that, The standardized JSON format record in step 4 includes timestamp, device_type, result, and status fields, where: ① The result field of pointer meters / multi-line digital meters is the reading + unit; ②The result field of the power cabinet is the path of the cabinet panel result image recognized by the YOLO model; ③ The result field of the infrared image is the path of the infrared temperature detection result image.

10. A substation robot inspection and identification system, used to implement the substation robot inspection and identification method as described in any one of claims 1-9, characterized in that, include: An image acquisition and preprocessing module is used to implement step 1 of the substation robot inspection and identification method. A multimodal equipment classification module is used to implement step 2 of the substation robot inspection and identification method. A differential identification processing module, which is used to implement step 3 of the substation robot inspection and identification method; and The data integration and standardization storage module is used to implement step 4 of the substation robot inspection and identification method.

Images