A Visual Intelligent Inspection Method for Substations Based on Virtual-Reality Matching

Abstract

An embodiment of the present invention provides a visual intelligent inspection method for substations based on virtual-real matching, belonging to the field of substation inspection. The method generates a digital twin substation consistent with the real substation scene by obtaining the scene layout and equipment distribution of the real-scene substation. After obtaining the real-field image of the real substation, the corresponding virtual-scene image is obtained, and through further calculation, the virtual-scene analysis image and the corresponding real-scene analysis image are obtained. The equipment to be detected is classified, and then the corresponding real-scene images after classification are detected accordingly, and the detection results are sent out to remind the staff. This method can greatly improve the automation operation and maintenance level of the substation, and can effectively improve the efficiency and accuracy of substation defect detection.

Description

Technical Field

[0001] The present invention relates to the field of substation inspection, and particularly to a visual intelligent inspection method for substations based on virtual-real matching. Background Art

[0002] The quality of inspection is closely related to the safe production of substations. Currently, the substation inspection method is developing towards the direction of intelligence and less manpower (unmanned). Traditional substation inspections based on manual methods have deficiencies such as high labor intensity, low efficiency, and difficulty in coping with harsh natural environments. To solve the inspection problems in large power stations using automated equipment, intelligent inspection robots for substations have emerged. These robots are usually equipped with detection devices such as visible light cameras, infrared thermal imagers, and microphones, and assist or replace manual labor to automatically detect and warn of the status of substation equipment by manually setting task points.

[0003] In recent years, vision inspection solutions based on high-definition cameras have begun to be widely deployed in large substation scenarios. By directly deploying cameras at key positions of inspection tasks (such as core substation equipment like main transformers), the problem of dynamic planning of task points for inspection robots can be solved to a certain extent, but this method still has many deficiencies. First, current intelligent equipment (robots, high-definition cameras, etc.) mainly adopts fixed inspection routes or inspection points, and this simple working method is difficult to cope with the complex and diverse equipment types and environmental changes in large substations; second, current intelligent inspection systems only complete the identification of specific types of image defects, and it is difficult to accurately locate these defects, resulting in the fact that the intelligence level of the system still cannot meet the demand for completely replacing manual labor; finally, current defect detection algorithms based on deep learning require high-definition images as input, and it is often difficult to cope with problems such as image blurring caused by light, rainfall, or unclear objects caused by long-distance shooting. Summary of the Invention

[0004] The purpose of the embodiments of the present invention is to provide a visual intelligent inspection method for substations based on virtual-real matching. This method constructs a digital twin substation of the real substation, which can greatly improve the automated operation and maintenance level of the substation and effectively improve the efficiency and accuracy of substation defect detection.

[0005] To achieve the above purpose, the embodiments of the present invention provide a visual intelligent inspection method for substations based on virtual-real matching, and the method includes:

[0006] Obtain the layout and equipment distribution model of the real substation scene;

[0007] Generate a virtual basic model of the digital twin substation according to the obtained layout and equipment distribution model of the real substation scene;

[0008] Correspond and update the devices in the virtual basic model of the digital twin substation with the devices in the real substation to obtain the accurate digital twin substation;

[0009] Obtain the distribution of monitoring cameras in the real substation;

[0010] Add the positions corresponding to the monitoring cameras in the real substation in the digital twin substation as virtual inspection points;

[0011] Obtain virtual scene images consistent with the real scene images obtained by the monitoring cameras in the real substation at the virtual inspection points;

[0012] Save the obtained virtual scene images and generate virtual scene analysis images according to the corresponding relationship with the devices in the real substation;

[0013] Obtain the real scene images obtained by the monitoring cameras in the real substation;

[0014] Match the virtual scene images with the real scene images according to the image matching algorithm;

[0015] Convert the virtual scene analysis images into real scene analysis images according to the matching results from the virtual scene images to the real scene images;

[0016] Determine the devices in the real substation that need to be detected;

[0017] Obtain the smallest rectangular area containing the devices in the real substation that need to be detected in the real scene analysis image;

[0018] Obtain the real scene images corresponding to the smallest rectangular area containing the devices in the real substation that need to be detected obtained in the real scene analysis image;

[0019] Crop the corresponding real scene images to obtain the smallest rectangular area of the real scene images containing the devices in the real substation that need to be detected;

[0020] Perform defect detection associated with the device type on the cropped real scene images to obtain the defect detection results of the devices in the real substation that need to be detected.

[0021] Optionally, the method includes:

[0022] Obtain the defect detection results;

[0023] Save the defect detection results;

[0024] Send the defect detection results for the staff to view.

[0025] Optionally, corresponding and updating the devices in the virtual basic model of the digital twin substation with the devices of the real substation to obtain an accurate digital twin substation includes:

[0026] Obtain the weather and lighting environment conditions of the real substation;

[0027] Further update the scene of the digital twin substation according to the obtained weather and lighting environment conditions of the real substation to obtain a more accurate digital twin substation.

[0028] Optionally, corresponding and updating the devices in the virtual basic model of the digital twin substation with the devices of the real substation to obtain an accurate digital twin substation includes:

[0029] Obtain the scene layout and devices in the real substation;

[0030] Classify the obtained scene layout and devices in the real substation to determine whether the number of the scene layout and the devices is greater than a first preset threshold;

[0031] Restore the scene layout and the devices less than the first preset threshold to the corresponding positions in the digital twin substation more accurately;

[0032] Restore the scene layout and the devices greater than the first preset threshold to the corresponding positions in the digital twin substation randomly.

[0033] Optionally, matching the virtual scene image with the real scene image according to the image matching algorithm includes:

[0034] Obtain the real scene image captured by the monitoring camera in the real substation and the virtual scene image consistent with the real scene image;

[0035] Perform data preprocessing on the real scene image and the virtual scene image to obtain the depth maps of the real scene image and the virtual scene image;

[0036] Screen the near-view parts in the real scene image and the virtual scene image with depth maps less than a second preset threshold;

[0037] Retain the near-view parts in the real scene image and the virtual scene image with depth maps less than the second preset threshold;

[0038] Obtain the image features of the screened real scene image and virtual scene image;

[0039] Perform regional division on the image features of the real scene image and the virtual scene image;

[0040] Match the image features of the region of the virtual scene image and the region of the real scene image corresponding to the region of the virtual scene image through the Random Sample Consensus (RANSAC) algorithm, so as to match the virtual scene image and the real scene image corresponding to the image features.

[0041] Optionally, matching the virtual scene image with the real scene image according to the image matching algorithm includes:

[0042] Obtain the virtual scene image and the real scene image of each region after matching;

[0043] Send the virtual scene image and the real scene image after matching into the feature extraction network to extract the features of the virtual scene image and the real scene image;

[0044] Send the features of the virtual scene image and the real scene image after extraction into the correlation network to calculate the similarity between the virtual scene image and the real scene image after matching, so as to obtain the similarity score matrix of the virtual scene image and the real scene image;

[0045] Send the similarity score matrix into the optical flow estimation network to obtain the optical flow field of the transformation for fine alignment of the virtual scene image and the real scene image;

[0046] Send the similarity score matrix into the mask calculation network to obtain the matching degree mask;

[0047] Judge whether the matching degree mask of each region is greater than the third preset threshold;

[0048] Take the region of the virtual scene image and the region of the real scene image corresponding to the virtual scene image that are less than the third preset threshold as the new virtual scene image and real scene image, and return to execute the step of performing regional division on the image features of the real scene image and the virtual scene image until the matching degree mask of the virtual scene image and the real scene image of each region is greater than the third preset threshold;

[0049] Fuse multiple optical flow fields to complete the final matching of the virtual scene image to the real scene image.

[0050] Optionally, obtaining the minimum rectangular region containing the equipment of the real substation to be detected in the real scene parsing image includes:

[0051] Obtain the real scene parsing image containing the equipment of the real substation to be detected;

[0052] Convert the real-scene analysis image into a grayscale image;

[0053] Use the built-in function of cv2 to obtain connected components for the grayscale image;

[0054] Calculate the vertex coordinates of the minimum bounding rectangle containing the device for each connected component;

[0055] Crop the area included in the vertex coordinates to obtain the minimum rectangular area containing the devices of the real substation to be detected;

[0056] Crop the corresponding minimum rectangular area on the corresponding real-scene analysis image.

[0057] Optionally, perform defect detection related to the device type on the cropped real-scene image to obtain the defect detection results of the devices of the real substation to be detected, including:

[0058] Obtain the cropped real-scene image;

[0059] Judge whether the devices of the real substation to be detected conform to the full-position defect detection of the device or the fixed-position defect detection of the device;

[0060] In the case where it is judged that the devices of the real substation to be detected conform to the full-position defect detection of the device, use the trained object detection network to detect the defect positions and types of the cropped real-scene image, and obtain the defect detection results.

[0061] Optionally, judging whether the devices of the real substation to be detected conform to the full-position defect detection of the device or the fixed-position defect detection of the device includes:

[0062] In the case where it is judged that the devices of the real substation to be detected conform to the fixed-position defect detection of the device, obtain the sub-devices of the devices of the real substation to be detected;

[0063] Obtain the real-scene image corresponding to the cropped real-scene analysis image containing the sub-devices;

[0064] Use the pre-trained resnet18 network on ImageNet as the teacher network, and use the randomly initialized resnet18 network as the student network;

[0065] Train the student network so that the output prediction features of the student network for the same input image are similar to the output prediction features of the teacher network;

[0066] Input the real-scene image into the student network and the teacher network;

[0067] Obtain the feature maps of the intermediate layers and the final feature maps in the student network and in the teacher network;

[0068] Calculate the cosine similarities of the feature maps of the intermediate layers and the final feature maps of the student network and the teacher network respectively;

[0069] Weightedly sum the cosine similarity values of the three feature maps to obtain an outlier of the real-scene image containing the sub-device to be detected;

[0070] Determine whether the outlier is greater than the fourth preset threshold;

[0071] In the case where it is determined that the outlier is greater than the fourth preset threshold, it is determined that the sub-device in the real-scene image containing the sub-device to be detected is abnormal.

[0072] Optionally, saving the obtained virtual scene image and generating a virtual scene analysis image according to the corresponding relationship with the devices of the real substation includes:

[0073] Obtain the devices of the real substation;

[0074] Correspond different devices of the real substation to different semantic colors;

[0075] Correspond the devices in the virtual scene image to the devices of the real substation;

[0076] Set the devices in the virtual scene image to their corresponding semantic colors to obtain a virtual scene analysis image.

[0077] Through the above technical solution, a substation visual intelligent inspection method based on virtual-real matching provided by the present invention generates a digital twin substation consistent with the real substation scene by obtaining the scene layout and device distribution of the real-scene substation. After obtaining the real-field image of the real substation, the corresponding virtual scene image is obtained, and through further calculation, a virtual scene analysis image and a corresponding real-scene analysis image are obtained. Classify the devices to be detected, then perform corresponding detection on the corresponding real-scene images after classification, and send the detection results to remind the staff. This method can greatly improve the automation operation and maintenance level of the substation, and can effectively improve the efficiency and accuracy of substation defect detection.

[0078] Other features and advantages of the embodiments of the present invention will be described in detail in the subsequent specific implementation part. BRIEF DESCRIPTION OF THE DRAWINGS

[0079] The accompanying drawings are used to provide a further understanding of the embodiments of the present invention and constitute a part of the specification. Together with the following detailed implementation manners, they are used to explain the embodiments of the present invention, but do not constitute a limitation to the embodiments of the present invention. In the accompanying drawings:

[0080] Figure 1 is a flowchart of a visual intelligent inspection method for a substation based on virtual-real matching according to an embodiment of the present invention;

[0081] Figure 2 is a partial flowchart of a visual intelligent inspection method for a substation based on virtual-real matching according to an embodiment of the present invention;

[0082] Figure 3 is a flowchart of adding weather and light to a digital twin substation in a visual intelligent inspection method for a substation based on virtual-real matching according to an embodiment of the present invention;

[0083] Figure 4 is a flowchart of further generating a digital twin substation in a visual intelligent inspection method for a substation based on virtual-real matching according to an embodiment of the present invention;

[0084] Figure 5 is a flowchart of matching virtual scene images and real scene images in a visual intelligent inspection method for a substation based on virtual-real matching according to an embodiment of the present invention;

[0085] Figure 6 is a flowchart of determining the minimum rectangular area in a visual intelligent inspection method for a substation based on virtual-real matching according to an embodiment of the present invention;

[0086] Figure 7 is a flowchart of defect detection in a visual intelligent inspection method for a substation based on virtual-real matching according to an embodiment of the present invention;

[0087] Figure 8 is a flowchart of generating a virtual scene analysis image in a visual intelligent inspection method for a substation based on virtual-real matching according to an embodiment of the present invention;

[0088] Figure 9 are the real scene image (left), virtual scene image (middle), and virtual scene analysis image (right) of a substation in a visual intelligent inspection method for a substation based on virtual-real matching according to an embodiment of the present invention. Detailed implementation manners

[0089] The following will describe in detail the specific implementation manners of the embodiments of the present invention with reference to the accompanying drawings. It should be understood that the specific implementation manners described herein are only used to illustrate and explain the embodiments of the present invention, and are not used to limit the embodiments of the present invention.

[0090] Figure 1 is a flowchart of a substation visual intelligent inspection method based on virtual-real matching according to an embodiment of the present invention. Figure 9 is a real-scene image (left), virtual-scene image (middle), and virtual-scene analysis image (right) of a substation of a substation visual intelligent inspection method based on virtual-real matching according to an embodiment of the present invention. The inspection method may include:

[0091] In step S1, obtain the real substation scene layout and equipment distribution model.

[0092] In step S2, generate a virtual basic model of the digital twin substation according to the real substation scene layout and equipment distribution model.

[0093] In step S3, correspond and update the equipment in the virtual basic model of the digital twin substation with the equipment in the real substation to obtain an accurate digital twin substation.

[0094] In step S4, obtain the distribution of monitoring cameras in the real substation.

[0095] In step S5, add the positions corresponding to the monitoring cameras in the real substation in the digital twin substation as virtual inspection points.

[0096] In step S6, obtain a virtual scene image consistent with the real scene image obtained by the monitoring camera in the real substation at the virtual inspection point.

[0097] In step S7, save the obtained virtual scene image, and generate a virtual scene analysis image according to the corresponding relationship with the equipment in the real substation.

[0098] In step S8, obtain the real scene image obtained by the monitoring camera in the real substation;

[0099] In step S9, match the virtual scene image with the real scene image according to the image matching algorithm;

[0100] In step S10, convert the virtual scene analysis image into a real scene analysis image according to the matching result from the virtual scene analysis image to the real scene analysis image.

[0101] In step S11, determine the equipment of the real substation that needs to be detected.

[0102] In step S12, obtain the minimum rectangular area of the devices of the real substation to be detected in the real-scene analysis image.

[0103] In step S13, obtain the real-scene image corresponding to the minimum rectangular area containing the devices of the real substation to be detected obtained in the real-scene analysis image.

[0104] In step S14, crop the corresponding real-scene image to obtain the minimum rectangular area of the real-scene image containing the devices of the real substation to be detected.

[0105] In step S15, perform defect detection associated with the device type on the cropped real-scene image to obtain the defect detection result of the devices of the real substation to be detected.

[0106] When it is necessary to detect the equipment in a substation, it is necessary to obtain the scene layout and equipment distribution of the real substation, and then generate a digital twin substation according to the situation of the real substation. After generating the digital twin substation, the equipment in the digital twin substation can be further corresponded and updated with the equipment in the real substation to generate a more accurate digital twin substation. In this digital twin substation, each point of the equipment that needs to be detected is corresponding to each point of the equipment in the real substation one by one. After updating the digital twin substation, the distribution of the monitoring cameras in the real substation can be obtained, and then virtual inspection points can be added at the corresponding positions in the digital twin substation according to the distribution of the monitoring cameras in the real substation. After adding the virtual inspection points, the orientation of the virtual inspection points can be adjusted according to the angle change of the monitoring cameras to obtain a virtual scene image consistent with the real scene image obtained by the monitoring cameras. After obtaining the virtual scene image, the virtual scene image can be saved and a virtual scene analysis image can be generated according to the corresponding relationship between the virtual scene image and the real scene image. Different equipment in the real substation can correspond to different semantic colors, and then be reflected in the virtual scene analysis image. After obtaining the virtual scene analysis image, the real scene image obtained by the monitoring cameras in the real substation can be obtained. The obtained virtual scene image corresponds to the real scene image. However, when obtaining the virtual scene image, there will be some deviations between the virtual scene image and the real scene image. Therefore, it is necessary to match the virtual scene image and the real scene image through an image matching algorithm. The corresponding relationship between the virtual scene analysis image and the real scene analysis image is the same as the corresponding relationship between the virtual scene image and the real scene image. Therefore, after matching the virtual scene image and the real scene image, the virtual scene analysis image can be converted to obtain the real scene analysis image according to the relationship and result between the virtual scene image and the real scene image. After obtaining the real scene analysis image, the equipment of the real substation that needs to be detected can be determined. After determining the equipment of the real substation that needs to be detected, the smallest rectangular area containing the equipment that needs to be detected can be obtained in the real scene analysis image, and then the smallest rectangular area can be corresponded to the real scene image, and then the real scene image can be cropped to obtain the smallest rectangular area of the real scene image containing the equipment that needs to be detected. After cropping to obtain the real scene image containing the equipment that needs to be detected, defect detection associated with the equipment type is performed on the image to obtain the defect detection result of the equipment of the real substation that needs to be detected.

[0107] In one embodiment of the present invention, Figure 2 is a partial flowchart of a substation visual intelligent inspection method based on virtual-real matching according to one embodiment of the present invention. The method may include:

[0108] In step S16, a defect detection result is obtained.

[0109] In step S17, the defect detection result is saved.

[0110] In step S18, the defect detection results are sent for staff to review.

[0111] After obtaining the defect detection results of the equipment in the real substation that needs to be detected, the defect detection results can be saved and sent to the staff for review. After the defect detection results are saved, it is convenient to perform subsequent tasks such as inspection process visualization or inspection result query.

[0112] In one embodiment of the present invention, Figure 3 The flowchart of adding weather and light conditions to a digital twin substation in a visual intelligent inspection method of a substation based on virtual-real matching according to an embodiment of the present invention. Adding weather and light conditions to a digital twin substation may include:

[0113] In step S19, the weather and lighting environment conditions of the actual substation are obtained.

[0114] In step S20, the scene of the digital twin substation is further updated according to the acquired weather and lighting environment conditions of the real substation to obtain a more accurate digital twin substation.

[0115] When constructing a digital twin substation, if the influence of environmental factors such as weather and lighting of the real substation is not considered, there will be deviations between the virtual scene of the digital twin substation and the real scene of the real substation when the digital twin substation is generated, which will affect the subsequent alignment effect. Therefore, when constructing a digital twin substation, the current environmental conditions such as weather and lighting of the real substation are obtained, and then the digital twin substation is further rendered according to the obtained environmental conditions such as weather and lighting, so that the virtual scene of the digital twin substation is as close to the real scene in the surveillance camera in the real substation as possible, so that the digital twin substation is more in line with the real substation, so as to facilitate the subsequent alignment work.

[0116] In one embodiment of the present invention, Figure 4 The present invention is a flowchart of further generating a digital twin substation according to a visual intelligent inspection method of a substation based on virtual-real matching according to an embodiment of the present invention. Generating the digital twin substation may include:

[0117] In step S21, the scene layout and equipment in the real substation are obtained.

[0118] In step S22, the acquired scene layout and equipment in the real substation are classified to determine whether the number of scene layout and equipment is greater than a first preset threshold.

[0119] In step S23, the scene layout and equipment smaller than the first preset threshold are restored to the corresponding positions in the digital twin substation more precisely.

[0120] In step S24, the scene layout and equipment larger than the first preset threshold are randomly restored to the corresponding positions in the digital twin substation.

[0121] When generating a digital twin substation, according to the different attributes of the equipment and scene layout of the real substation, it can be accurately or vaguely corresponding to the digital twin substation. When restoring the real substation to the digital twin substation in a 1:1 ratio, the objects such as the scene layout and equipment in the real substation can be classified according to material attributes, such as land, roads, trees, transformers, etc. The scene layout can be objects with a large number and high repeatability, such as land and trees. Different construction methods can be adopted for objects with different attributes. When the quantity and repeatability of the obtained scene layout and equipment are less than the first preset threshold, it indicates that the obtained scene layout and equipment should be objects with a small quantity and low repeatability, such as substation equipment. Through steps such as creating vertices, drawing triangles, drawing color maps, creating model materials, and adding light and shadow, a three-dimensional model of the equipment of the real substation can be accurately constructed, so that each object has an accurately restored virtual three-dimensional model. When the quantity and repeatability of the obtained scene layout and equipment are greater than the first preset threshold, it indicates that the obtained scene layout and equipment should be objects with a large number and high randomness, such as land and trees. Therefore, when constructing a virtual three-dimensional model for it, it can be randomly generated at the corresponding position in the digital twin substation, and the random attributes can include rotation, scaling, small displacement, etc.

[0122] In an embodiment of the present invention, as Figure 5 shown, the matching of the virtual scene image and the real scene image may include:

[0123] In step S25, obtain the real scene image captured by the monitoring camera in the real substation and the virtual scene image consistent with the real scene image.

[0124] In step S26, perform data preprocessing on the real scene image and the virtual scene image to obtain the depth maps of the real scene image and the virtual scene image.

[0125] In step S27, filter out the foreground parts in the real scene image and the virtual scene image with depth maps less than the second preset threshold.

[0126] In step S28, retain the foreground parts in the real scene image and the virtual scene image with depth maps less than the second preset threshold.

[0127] In step S29, the image features of the filtered real scene image and virtual scene image are obtained.

[0128] In step S30, the image features of the real scene image and virtual scene image are divided into regions.

[0129] In step S31, the image features of the regions of the virtual scene image and the regions of the real scene image corresponding to the regions of the virtual scene image are matched by the random sample consensus algorithm, so as to match the virtual scene image and the real scene image corresponding to the image features.

[0130] Although a real scene image and a virtual scene image consistent with the real scene image can be obtained, due to problems such as camera angle deviation in a real substation, there may be deviations between the real scene image and the virtual scene image, and a matching relationship between the virtual scene image and the real scene image needs to be obtained. Therefore, the virtual scene image and the real scene image need to be matched. During the matching process, the obtained real scene image and virtual scene image can be preprocessed to obtain depth maps of the real scene image and the virtual scene image. Then, the foreground parts in the real scene image and virtual scene image with depth map information less than a second preset threshold are filtered. According to the second preset threshold, the virtual scene image and the real scene image can be divided into a foreground part at a short distance and a background part at a long distance, and then the background parts in the virtual scene image and the real scene image are removed to avoid interference from meaningless scene content in the far distance to the matching result. After obtaining the filtered real scene image and virtual scene image, the image features of the real scene image and the virtual scene image are obtained again, and then the image features in the real scene image and the virtual scene image are divided into regions. The image features of the virtual scene image and the image features of the real scene image after region division are matched by the random sample consensus algorithm, so as to match the virtual scene image and the real scene image corresponding to the image features. This is to facilitate further analysis of the matched virtual scene image and real scene image in the subsequent process.

[0131] In an embodiment of the present invention, as Figure 5 shown, after step S31, it may further include:

[0132] In step S32, the real scene image and the virtual scene image of each matched region are obtained.

[0133] In step S33, the matched real scene image and virtual scene image are sent to a feature extraction network to extract the features of the real scene image and the virtual scene image.

[0134] In step S34, the features of the extracted real scene image and virtual scene image are fed into the correlation network to calculate the similarity between the matched real scene image and virtual scene image, so as to obtain a similarity score matrix for the real scene image and virtual scene image.

[0135] In step S35, the similarity score matrix is fed into the optical flow estimation network to obtain the optical flow field of the transformation for the fine alignment of the virtual scene image and the real scene image.

[0136] In step S36, the similarity score matrix is fed into the mask calculation network to obtain a matching degree mask.

[0137] In step S37, it is judged whether the matching degree mask of each region is greater than a third preset threshold.

[0138] In step S38, the regions of the virtual scene image and the corresponding real scene image with the matching degree mask less than the third preset threshold are used as the new virtual scene image and real scene image, and returned to step S30 until the matching degree masks of the virtual scene image and the real scene image in each region are greater than the third preset threshold.

[0139] In step S39, multiple optical flow fields are fused to complete the final matching of the virtual scene image and the real scene image.

[0140] After step S31, the virtual scene image and the real scene image have been matched. However, at this time, the virtual scene image and the real scene image still belong to rough matching, and the matching effect between the virtual scene image and the real scene image is still insufficient. Therefore, the matched virtual scene image and real scene image need to be finely aligned again. The matched virtual scene image and real scene image are sent into a feature extraction network to obtain the features of the virtual scene image and the real scene image. The features of the virtual scene image and the real scene image are sent into a correlation network to calculate the similarity between the matched virtual scene image and the real scene image, so as to obtain a similarity score matrix of the virtual scene image and the real scene image. Sending this similarity score matrix into an optical flow estimation network can obtain an optical flow field of the transformation for the fine alignment of the virtual scene image and the real scene image. Sending this similarity score matrix into a mask calculation network to obtain a matching degree mask. This matching degree mask can measure whether the effect after the fine alignment of the virtual scene image and the real scene image in each region meets the requirements. If the matching degree mask of a region is less than the third preset threshold, the alignment effect of the virtual scene image and the real scene image in this region is poor, and then the virtual scene image and the real scene image in this region need to be sent back to step S30 as new virtual scene images and real scene images for cycling until the matching degree mask of the virtual scene image and the real scene image in each region is greater than the third preset threshold, indicating that the effect after the fine alignment of the virtual scene image and the real scene image in each region has met the requirements. At this time, the optical flow fields of each region after fine alignment are fused to complete the final matching of the virtual scene image and the real scene image.

[0141] After fusing the optical flow field, the optical flow field can be applied to the virtual scene parsing image, so that the virtual scene parsing image can be converted to obtain a real scene parsing image.

[0142] In an embodiment of the present invention, as Figure 6 shown, obtaining the minimum rectangular region may include:

[0143] In step S40, a real scene parsing image of the device including the real substation to be detected is obtained.

[0144] In step S41, the real scene parsing image is converted into a grayscale image.

[0145] In step S42, the built-in function of cv2 is used to obtain connected components for the grayscale image.

[0146] In step S43, the vertex coordinates of the minimum bounding rectangle containing the device are calculated for each connected component.

[0147] In step S44, the region included in the vertex coordinates is clipped to obtain the smallest rectangular region that includes the devices of the real substation to be detected.

[0148] In step S45, the corresponding smallest rectangular region is clipped on the corresponding real-scene analysis image.

[0149] When detecting defects in the devices of the substation, it is necessary to determine the device to be detected in the virtual-scene image, then determine the corresponding device in the virtual-scene analysis image, and determine the corresponding device in the real-scene analysis image through the conversion relationship between the virtual-scene analysis image and the real-scene analysis image. Then, the real-scene analysis image containing the device is converted into a grayscale image, and the built-in function cv2.connectedComponents in cv2 is used to obtain the connected regions for this grayscale image. A series of calculations are performed on this connected region to obtain the smallest rectangular region that includes the device to be detected, and the corresponding smallest rectangular region is clipped on the real-scene analysis image.

[0150] In one embodiment of the present invention, as Figure 7 shown, the defect detection of the devices of the real substation may include:

[0151] In step S46, the clipped real-scene image is obtained.

[0152] In step S47, it is determined whether the clipped real-scene image conforms to the defect detection of all positions of the device or the defect detection of fixed positions of the device.

[0153] In step S48, in the case where it is determined that the clipped real-scene image conforms to the defect detection of all positions of the device, the trained target detection network is used to detect the defect positions and obtain the defect detection results.

[0154] After obtaining the clipped real-scene image, it is determined whether the devices of the substation to be detected conform to the defect detection of all positions of the device or the defect detection of fixed positions of the device. In the case where the devices of the substation to be detected conform to the defect detection of all positions of the device, it indicates that the defect may appear in all positions of the device, and the defect may be rust, oil stain, bird's nest, etc. At this time, the pre-trained target detection network such as yolox can be used to detect the defect positions and types of this real-scene image, and the detection results can be obtained.

[0155] In one embodiment of the present invention, as Figure 7 shown, the defect detection of the devices of the real substation may also include:

[0156] In step S49, when it is determined that the equipment of the real substation to be detected meets the equipment fixed - position defect detection, the sub - equipment of the equipment of the real substation to be detected is obtained.

[0157] In step S50, the real - scene image corresponding to the parsed real - scene image containing the sub - equipment after cropping is obtained.

[0158] In step S51, the pre - trained resnet18 network on ImageNet is used as the teacher network, and the randomly initialized resnet18 network is used as the student network.

[0159] In step S52, the student network is trained so that the output prediction features of the student network for the same input image are similar to the output prediction features of the teacher network.

[0160] In step S53, the real - scene image is input into the student network and the teacher network.

[0161] In step S54, the cosine similarity of the feature maps in the intermediate layer and the final feature maps in the student network and the teacher network is obtained.

[0162] In step S55, the cosine similarity values of the three feature maps are weighted and added together to obtain the outlier value of the real - scene image containing the sub - equipment to be detected.

[0163] In step S56, it is judged whether the outlier value is greater than the fourth preset threshold.

[0164] In step S57, when it is judged that the outlier value is greater than the fourth preset threshold, it is determined that the sub - equipment in the real - scene image containing the sub - equipment to be detected is abnormal.

[0165] When it is determined that the equipment of the real substation to be detected meets the equipment all - position defect detection, it indicates that the defect may appear in the fixed position of the equipment, such as water ingress in the instrument panel, insulator breakage, etc. Therefore, ImageNet can be used to identify the sub - equipment at the fixed position of the equipment. The real - scene image containing the sub - equipment is input into the classroom network and the student network, and whether the sub - equipment is abnormal is judged according to the features output by the teacher network and the student network. The cosine similarity values of the three feature maps are weighted and added together, so that the defect of the sub - equipment to be detected can be evaluated. Therefore, the cosine similarity values of the three are weighted and added together to obtain the outlier value of the real - scene image containing the sub - equipment to be detected, and this outlier value is judged. When it is judged that this outlier value is greater than the fourth preset threshold, it indicates that the sub - equipment is abnormal.

[0166] In one embodiment of the present invention, as Figure 8 shown, generating the virtual - scene parsed image may include:

[0167] In step S58, the devices of the real substation are obtained.

[0168] In step S59, different colors are assigned to different devices of the real substation.

[0169] In step S60, the devices in the virtual scene image are corresponded to the devices of the real substation.

[0170] In step S61, the devices in the virtual scene image are set to their corresponding colors to obtain the virtual scene analysis image.

[0171] Before defect detection of the lacking devices, it is necessary to determine the objects to be detected in the virtual scene analysis image, and then determine the devices to be detected in the real scene analysis image according to the corresponding relationship with the virtual scene analysis image. Therefore, it is necessary to obtain the virtual scene analysis image first. When generating a digital twin substation, each device in the real substation is numbered. Therefore, different devices in the real substation can be obtained by acquiring different numbers, and then different semantic colors are set for different devices in the substation. Different devices in the virtual scene image are set to different semantic colors to obtain the scene analysis image. Therefore, when it is necessary to detect the devices in the substation, it can be achieved by finding the semantic colors corresponding to the devices in the virtual scene analysis image.

[0172] Through the above technical solution, a substation visual intelligent inspection method based on virtual-real matching provided by the present invention generates a digital twin substation consistent with the real substation scene by obtaining the scene layout and device distribution of the real-scene substation. After obtaining the real-field image of the real substation, the corresponding virtual scene image is obtained, and through further calculation, the virtual scene analysis image and the corresponding real scene analysis image are obtained. The devices to be detected are classified, and then the corresponding real scene images after classification are detected accordingly, and the detection results are sent out to remind the staff. This method can greatly improve the automated operation and maintenance level of the substation, and can effectively improve the efficiency and accuracy of substation defect detection.

[0173] It should also be noted that the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements not only includes those elements, but also includes other elements not expressly listed, or also includes elements inherent to such process, method, commodity or device. Without further limitation, an element defined by the statement "including one..." does not exclude the existence of another identical element in the process, method, commodity or device including the element.

[0174] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, various modifications and changes can be made to the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included within the scope of the claims of the present application.

Claims

1. A visual intelligent inspection method for substations based on virtual-real matching, characterized in that, The method includes: Obtain the layout and equipment distribution model of the actual substation scenario; Generate a virtual basic model of the digital twin substation based on the obtained layout and equipment distribution model of the actual substation scenario; Correspond and update the equipment in the virtual basic model of the digital twin substation with the equipment in the actual substation to obtain the accurate digital twin substation; Obtain the distribution of monitoring cameras in the actual substation; Add the positions corresponding to the monitoring cameras in the actual substation in the digital twin substation as virtual inspection points; Obtain virtual scenario images consistent with the real scenario images obtained by the monitoring cameras in the actual substation at the virtual inspection points; Save the obtained virtual scenario images and generate virtual scenario analysis images according to the correspondence with the equipment in the actual substation; Obtain the real scenario images obtained by the monitoring cameras in the actual substation; Match the virtual scenario images with the real scenario images according to the image matching algorithm; Convert the virtual scenario analysis images into real scenario analysis images according to the matching results from the virtual scenario images to the real scenario images; Determine the equipment in the actual substation to be detected; Obtain the smallest rectangular area in the real scenario analysis image that contains the equipment in the actual substation to be detected; Obtain the real scenario image corresponding to the smallest rectangular area that contains the equipment in the actual substation to be detected obtained in the real scenario analysis image; Crop the corresponding real scenario image to obtain the smallest rectangular area of the real scenario image that contains the equipment in the actual substation to be detected; Perform defect detection associated with the equipment type on the cropped real scenario image to obtain the defect detection results of the equipment in the actual substation to be detected.

2. The method according to claim 1, characterized in that, The method includes: Obtain the defect detection results; Save the defect detection results; Send the defect detection results for the staff to view.

3. The method according to claim 1, characterized in that, Corresponding and updating the equipment in the virtual basic model of the digital twin substation with the equipment in the actual substation to obtain an accurate digital twin substation includes: Obtain the weather and lighting environment conditions of the actual substation; Further update the scenario of the digital twin substation according to the obtained weather and lighting environment conditions of the actual substation to obtain a more accurate digital twin substation.

4. The method according to claim 1, characterized in that, Corresponding and updating the equipment in the virtual basic model of the digital twin substation with the equipment in the actual substation to obtain an accurate digital twin substation includes: Obtain the scenario layout and equipment in the actual substation; Classify the obtained scenario layout and equipment in the actual substation to determine whether the quantity of the scenario layout and the equipment is greater than a first preset threshold; Restore the scenario layout and the equipment less than the first preset threshold more accurately to the corresponding positions in the digital twin substation; Restore the scenario layout and the equipment greater than the first preset threshold randomly to the corresponding positions in the digital twin substation.

5. The method according to claim 1, characterized in that, Matching the virtual scene image with the real scene image according to the image matching algorithm includes: Obtaining the real scene image acquired by the monitoring camera in the real substation and the virtual scene image consistent with the real scene image; Performing data preprocessing on the real scene image and the virtual scene image to obtain the depth maps of the real scene image and the virtual scene image; Screening the foreground parts in the real scene image and the virtual scene image with depth maps less than the second preset threshold; Retaining the foreground parts in the real scene image and the virtual scene image with depth maps less than the second preset threshold; Obtaining the image features of the screened real scene image and virtual scene image; Performing regional division on the image features of the real scene image and the virtual scene image; Matching the image features of the region of the virtual scene image and the region of the real scene image corresponding to the region of the virtual scene image through the random sample consensus algorithm to match the virtual scene image and the real scene image corresponding to the image features.

6. The method according to claim 5, wherein, Matching the virtual scene image with the real scene image according to the image matching algorithm includes: Obtaining the virtual scene image and the real scene image of each region after matching; Sending the virtual scene image and the real scene image after matching into the feature extraction network to extract the features of the virtual scene image and the real scene image; Feeding the features of the virtual scene image and the real scene image after extraction into the correlation network to calculate the similarity between the virtual scene image and the real scene image after matching, so as to obtain the similarity score matrix of the virtual scene image and the real scene image; Sending the similarity score matrix into the optical flow estimation network to obtain the optical flow field of the transformation for fine alignment of the virtual scene image and the real scene image; Sending the similarity score matrix into the mask calculation network to obtain the matching mask; Judging whether the matching mask of each region is greater than the third preset threshold; Taking the region of the virtual scene image and the region of the real scene image corresponding to the virtual scene image that are less than the third preset threshold as the new virtual scene image and real scene image, and returning to execute the step of performing regional division on the image features of the real scene image and the virtual scene image until the matching mask of the virtual scene image and the real scene image of each region is greater than the third preset threshold; Fusing multiple optical flow fields to complete the final matching of the virtual scene image to the real scene image.

7. The method according to claim 1, wherein, Obtaining the smallest rectangular region containing the equipment of the real substation to be detected in the real scene analysis image includes: Obtaining the real scene analysis image containing the equipment of the real substation to be detected; Converting the real scene analysis image into a grayscale image; Using the built-in function of cv2 to obtain the connected components of the grayscale image; Calculating the vertex coordinates of the smallest bounding rectangle containing the equipment for each connected component; Clip the area included in the vertex coordinates to obtain the smallest rectangular area that includes the devices of the real substation to be detected; Clip the corresponding smallest rectangular area on the corresponding real scene analysis image.

8. The method according to claim 1, wherein, Perform defect detection associated with the device type on the clipped real scene image to obtain the defect detection results of the devices of the real substation to be detected, including: Obtain the clipped real scene image; Judge whether the devices of the real substation to be detected conform to the all-position defect detection of the device or the fixed-position defect detection of the device; In the case where it is judged that the devices of the real substation to be detected conform to the all-position defect detection of the device, use the trained object detection network to detect the defect positions and types of the clipped real scene image, and obtain the defect detection results.

9. The method according to claim 8, wherein, Judging whether the devices of the real substation to be detected conform to the all-position defect detection of the device or the fixed-position defect detection of the device includes: In the case where it is judged that the devices of the real substation to be detected conform to the fixed-position defect detection of the device, obtain the sub-devices of the devices of the real substation to be detected; Obtain the real scene image corresponding to the real scene analysis image including the sub-devices after clipping; Use the ImageNet pre-trained resnet18 network as the teacher network, and use the randomly initialized resnet18 network as the student network; Train the student network so that the output prediction features of the student network for the same input image are similar to the output prediction features of the teacher network; Input the real scene image into the student network and the teacher network; Obtain the feature maps of the intermediate layer and the final feature maps in the student network and the teacher network; Calculate the cosine similarities of the feature maps of the intermediate layer and the final feature maps of the student network and the teacher network respectively; Weightedly sum the cosine similarity values of the three feature maps to obtain the outlier value of the real scene image including the sub-devices to be detected; Judge whether the outlier value is greater than the fourth preset threshold; In the case where it is judged that the outlier value is greater than the fourth preset threshold, determine that the sub-devices in the real scene image including the sub-devices to be detected are abnormal.

10. The method according to claim 1, wherein, Saving the obtained virtual scene image and generating a virtual scene analysis image according to the corresponding relationship with the devices of the real substation includes: Obtain the devices of the real substation; Assign different semantic colors to different devices of the real substation; Correspond the devices in the virtual scene image with the devices of the real substation; Set the devices in the virtual scene image to their corresponding semantic colors to obtain the virtual scene analysis image.

Images