Defect target detection method and device for robot fish operation inside transformer

Abstract

The present invention provides a defect target detection method and device for robot fish operation inside a transformer, which can be applied to the field of image processing technology. Specifically, the defect target detection method includes: in response to a transformer internal state detection request, obtaining an internal structure image of the transformer collected by a robot fish inside the transformer; based on the internal structure image, determining the quality information of the transformer oil inside the transformer; based on the quality information of the transformer oil, correcting the internal structure image to obtain a first image; performing object detection based on the first image to obtain a defect target detection result of the transformer.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing technology, and more specifically, to a defect target detection method and device for a robot fish operation inside a transformer. Background Art

[0002] As a key equipment in the power system, the operating status of large transformers is directly related to the stability and safety of the power grid. However, due to the complex internal structure of the transformer and the long-term operation in harsh environments such as high voltage, high temperature, and high humidity, various defects are prone to occur inside the transformer. With the rapid development of robotics technology, more and more robots are replacing humans to detect the internal status of transformers to improve detection efficiency and accuracy.

[0003] However, due to the long-term influence of high voltage, high temperature and electromagnetic field inside the transformer, the transformer oil deteriorates, discolors, and accumulates a large amount of impurities. These interference factors cause the captured images to have low contrast, color distortion, poor clarity, etc., increasing the risk of false detection and missed detection. In addition, due to the complex internal structure of the transformer, the light source is difficult to distribute evenly, resulting in uneven brightness in the image. And due to the small space inside the transformer, the light source carried by the robot may not be able to fully illuminate all areas, further exacerbating the lighting problem. Summary of the invention

[0004] In view of this, the present invention provides a defect target detection method and device for robot fish operation inside a transformer.

[0005] One aspect of the present invention provides a defect target detection method for robot fish operation inside a transformer, comprising: in response to a transformer internal state detection request, obtaining an internal structure image of the transformer collected by the robot fish inside the transformer; based on the internal structure image, determining quality information of the transformer oil inside the transformer; based on the quality information of the transformer oil, correcting the internal structure image to obtain a first image; performing object detection based on the first image to obtain a defect target detection result of the transformer.

[0006] According to an embodiment of the present invention, the object detection is performed based on the first image to obtain the defect target detection result of the transformer, including: performing image enhancement processing on the first image based on adaptive histogram equalization and a self-calibrated illumination learning network to obtain an enhanced image; performing super-resolution reconstruction on the enhanced image to obtain a reconstructed image; and performing object detection and instance segmentation on the reconstructed image to obtain the defect target detection result of the transformer.

[0007] According to an embodiment of the present invention, the image enhancement processing based on adaptive histogram equalization and self-calibration illumination learning network is performed on the first image to obtain an enhanced image, including: using a histogram equalization method to perform contrast enhancement on the first image to obtain a grayscale image; splicing the grayscale image with the first image according to the channel dimension to obtain a second image; performing multi-scale feature extraction on the second image to obtain a first feature map; inputting the first feature map into a self-calibration illumination learning network to obtain a second feature map; performing multi-scale fusion on the second feature map and the grayscale image to obtain a third feature map; using a residual network to process the third feature map, and outputting the enhanced image.

[0008] According to an embodiment of the present invention, the super-resolution reconstruction of the enhanced image to obtain the reconstructed image includes: inputting the enhanced image into a generator of an adversarial network to perform super-resolution reconstruction on the enhanced image to obtain the reconstructed image.

[0009] According to an embodiment of the present invention, the above-mentioned object detection and instance segmentation are performed on the above-mentioned reconstructed image to obtain the defect target detection result of the above-mentioned transformer, including: inputting the above-mentioned reconstructed image into an object detection network to obtain a target detection feature map; performing feature extraction on the above-mentioned reconstructed image to obtain a fourth feature map; using an attention mechanism to fuse the above-mentioned target detection feature map and the above-mentioned fourth feature map to obtain a cross feature map; inputting the above-mentioned cross feature map into an instance segmentation network to obtain a pixel mask related to the detection target; based on the above-mentioned pixel mask, obtaining the defect target detection result of the above-mentioned transformer.

[0010] According to an embodiment of the present invention, the quality information of the transformer oil inside the transformer is determined based on the internal structure image, including: extracting a first sub-image containing transformer components and a second sub-image containing background information from the internal structure image; determining the distortion information of the transformer oil based on the shape features of the transformer components in the first sub-image and the actual shape features of the transformer structure; performing abnormal pixel point detection on the second sub-image to determine at least one first abnormal pixel block, wherein the first abnormal pixel block is composed of a plurality of abnormal pixel points; obtaining the color information of the transformer oil based on the pixel information of other pixel points in the second sub-image except the at least one first abnormal pixel block; determining the impurity information of the transformer oil based on the number of abnormal pixel points contained in each of the at least one first abnormal pixel blocks, wherein the impurity information is used to represent the size probability distribution of the abnormal pixel blocks.

[0011] According to an embodiment of the present invention, the above-mentioned internal structure image is corrected based on the quality information of the above-mentioned transformer oil to obtain a first image, including: performing abnormal pixel point detection on the above-mentioned internal structure image to determine a plurality of second abnormal pixel blocks; determining the confidence of each of the above-mentioned plurality of second abnormal pixel blocks based on the number of abnormal pixel points contained in each of the above-mentioned plurality of second abnormal pixel blocks and the size probability distribution of the abnormal pixel blocks represented by the above-mentioned impurity information; determining at least one target abnormal pixel block from the above-mentioned plurality of second abnormal pixel blocks based on a confidence threshold; eliminating at least one target abnormal pixel block in the above-mentioned internal structure image to obtain a third image; performing color correction on the above-mentioned third image based on the above-mentioned color information to obtain a fourth image; and correcting the above-mentioned fourth image based on the above-mentioned distortion information to obtain the above-mentioned first image.

[0012] According to an embodiment of the present invention, the method further includes: performing data enhancement of initial samples based on multiple defect types of the transformer to obtain multiple training samples; and using the multiple training samples to train a self-calibrated illumination learning network, an adversarial network, an object detection network, and an instance segmentation network.

[0013] According to an embodiment of the present invention, the data enhancement of the initial sample based on the multiple defect types of the above-mentioned transformer is performed to obtain multiple training samples, including: for each of the above-mentioned defect types, extracting a first defect target image corresponding to the above-mentioned defect type from a defect library; randomly scaling the above-mentioned first defect target image to obtain a second defect target image; and superimposing the above-mentioned second defect target image on the above-mentioned initial sample to obtain a training sample corresponding to the above-mentioned defect type.

[0014] Another aspect of the present invention provides a defect target detection device for robot fish operation inside a transformer, comprising: an acquisition module, used to respond to a transformer internal state detection request and acquire an internal structure image of the transformer collected by the robot fish inside the transformer; a determination module, used to determine the quality information of the transformer oil inside the transformer based on the internal structure image; a correction module, used to correct the internal structure image based on the quality information of the transformer oil to obtain a first image; and a detection module, used to perform object detection based on the first image to obtain a defect target detection result of the transformer.

[0015] Another aspect of the present invention provides an electronic device, comprising: one or more processors; and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method described above.

[0016] Another aspect of the present invention provides a computer-readable storage medium storing computer-executable instructions, which are used to implement the above method when executed.

[0017] Another aspect of the present invention provides a computer program product, the computer program product comprising computer executable instructions, and the instructions are used to implement the above method when executed.

[0018] According to the embodiment of the present invention, by analyzing the transformer oil image features in the transformer internal structure image, the quality information of the transformer oil can be accurately evaluated, thereby helping to timely discover the aging and pollution of the oil and other problems, and providing an important basis for the maintenance and management of the transformer. In addition, the object detection model is used to perform feature extraction and other processing on the corrected internal structure image to improve the detection accuracy of defective targets inside the transformer and better judge the operating status of the transformer. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The above and other objects, features and advantages of the present invention will become more apparent through the following description of the embodiments of the present invention with reference to the accompanying drawings.

[0020] Figure 1 An exemplary system architecture of a defect target detection method for robot fish operation inside a transformer according to an embodiment of the present invention is shown.

[0021] Figure 2 A flow chart of a defect target detection method for a robot fish operation inside a transformer according to an embodiment of the present invention is shown.

[0022] Figure 3 A flow chart of an object detection and instance segmentation method according to an embodiment of the present invention is shown.

[0023] Figure 4 A block diagram of a defect target detection device for robot fish operation inside a transformer according to an embodiment of the present invention is shown.

[0024] Figure 5 A block diagram of an electronic device suitable for implementing a defect target detection method for a robot fish operation inside a transformer according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0025] Below, embodiments of the present invention will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present invention. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of embodiments of the present invention. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of concepts of the present invention.

[0026] The terms used herein are only for describing specific embodiments and are not intended to limit the present invention. The terms "comprise", "include", etc. used herein indicate the existence of the features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0027] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0028] When using expressions such as "at least one of A, B, and C, etc.", they should generally be interpreted according to the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0029] In the embodiments of the present invention, the collection, updating, analysis, processing, use, transmission, provision, invention, storage and other aspects of the data involved (for example, including but not limited to user personal information) are in compliance with the provisions of relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. In particular, necessary measures are taken for user personal information to prevent illegal access to user personal information data and maintain the security of user personal information, network security and national security.

[0030] In the embodiment of the present invention, before obtaining or collecting the user's personal information, the user's authorization or consent is obtained.

[0031] As a key equipment of the power system, the operating status of large transformers is directly related to the stability and safety of the power grid. However, due to the complex internal structure of the transformer and the long-term operation in harsh environments such as high voltage, high temperature, and high humidity, various defects are prone to occur inside the transformer, such as insulation aging, winding deformation, oil deterioration, etc. If these defects are not discovered and handled in time, they may cause serious power accidents, resulting in huge economic losses and social impacts.

[0032] Traditional methods of detecting the internal status of large transformers mainly rely on manual drilling and cover inspection, which not only consumes a lot of manpower and material resources, but also poses safety risks. In addition, due to the limitations of manual inspection, it is difficult to ensure the accuracy and comprehensiveness of the inspection.

[0033] Therefore, with the rapid development of robotics technology, relevant researchers have begun to explore the use of robots to replace humans in transformer internal status detection to improve detection efficiency and accuracy.

[0034] However, due to the long-term influence of high voltage, high temperature and electromagnetic field inside the transformer, the transformer oil deteriorates, discolors, and accumulates a large amount of impurities. The impurities and oil accumulated inside the transformer will form noise or occlusion in the image, interfering with the identification of defects. These interference factors cause the captured images to have low contrast, color distortion, poor clarity and other problems, increasing the risk of false detection and missed detection. In addition, due to the complex internal structure of the transformer, the light source is difficult to distribute evenly, resulting in uneven brightness in the image, and due to the small internal space of the transformer, the light source carried by the robot may not be able to fully illuminate all areas, further exacerbating the lighting problem.

[0035] In view of this, the embodiments of the present invention can accurately evaluate the quality information of transformer oil by analyzing the transformer oil image features in the transformer internal structure image, thereby helping to timely discover the aging and pollution problems of the oil, and providing an important basis for the maintenance and management of the transformer. In addition, the object detection model is used to perform feature extraction and other processing on the corrected internal structure image to improve the detection accuracy of defective targets inside the transformer and better judge the operating status of the transformer.

[0036] Specifically, an embodiment of the present invention provides a defect target detection method for robot fish operations inside a transformer, including: in response to a transformer internal state detection request, obtaining an internal structure image of the transformer collected by a robot fish inside the transformer; based on the internal structure image, determining quality information of the transformer oil inside the transformer; based on the quality information of the transformer oil, correcting the internal structure image to obtain a first image; performing object detection based on the first image to obtain a defect target detection result of the transformer.

[0037] Figure 1 The exemplary system architecture of the defect target detection method for the robot fish operation inside the transformer according to the embodiment of the present invention is shown. It should be noted that Figure 1 The examples shown are merely examples of system architectures to which the embodiments of the present invention may be applied, to help those skilled in the art understand the technical content of the present invention, but do not mean that the embodiments of the present invention may not be used in other devices, systems, environments or scenarios.

[0038] like Figure 1 As shown, the system architecture according to this embodiment may include a transformer 1, a transformer internal inspection device 2, and a processing device 3. A medium such as a communication link such as a network is provided between the transformer internal inspection device 2 and the processing device 3. The network may include various connection types, such as wired and / or wireless communication links, etc.

[0039] The transformer internal inspection device 2 may be located inside the transformer 1, and may interact with the processing device 3 through a network to receive or send information, etc. For example, the transformer internal inspection device 2 may be a device such as an internal inspection robot fish.

[0040] The processing device 3 can be set at any position outside the transformer 1. The processing device 3 can be a server that provides various services to analyze and process the data such as the internal structure image of the transformer collected by the robot fish inside the transformer.

[0041] It should be noted that the defect target detection method for robot fish operation inside the transformer provided in the embodiment of the present invention can generally be executed by the processing device 3. Correspondingly, the defect target detection device for robot fish operation inside the transformer provided in the embodiment of the present invention can generally be arranged in the processing device 3. The defect target detection method for robot fish operation inside the transformer provided in the embodiment of the present invention can also be executed by a server or server cluster that is different from the processing device 3 and can communicate with the transformer internal inspection device 2 and / or the processing device 3. Correspondingly, the defect target detection device for robot fish operation inside the transformer provided in the embodiment of the present invention can also be arranged in a server or server cluster that is different from the processing device 3 and can communicate with the transformer internal inspection device 2 and / or the processing device 3.

[0042] For example, the internal structure image of the transformer can be collected by the transformer internal inspection device 2, and then the transformer internal inspection device 2 can send the internal structure image of the transformer to the processing device 3, and the processing device 3 that receives the internal structure image of the transformer can execute the defect target detection method for the robot fish operation inside the transformer provided by the embodiment of the present invention.

[0043] It should be understood that Figure 1 The number of transformer internal inspection devices, networks and processing devices in the figure is only for illustration. Any number of internal inspection devices, networks and processing devices may be provided according to implementation requirements.

[0044] Figure 2 A flow chart of a defect target detection method for a robot fish operation inside a transformer according to an embodiment of the present invention is shown.

[0045] like Figure 2 As shown, the method includes operations S201 to S204.

[0046] In operation S201 , in response to a transformer internal state detection request, an internal structure image of the transformer captured by a robotic fish inside the transformer is acquired.

[0047] According to an embodiment of the present invention, in response to a transformer internal state detection request, a robot fish deployed inside the transformer is used to collect an internal structure image of the transformer. The internal structure image of the transformer may include structural images of key components such as the core, winding, and oil tank wall, and images of the oil environment, so as to determine whether there are impurities, defects, or abnormal objects inside the transformer.

[0048] In operation S202 , quality information of transformer oil inside the transformer is determined based on the internal structure image.

[0049] According to an embodiment of the present invention, transformer oil image features related to transformer oil, such as oil color, transparency, presence of suspended matter or sediment, etc., can be extracted from the collected internal structure image of the transformer. Based on the extracted transformer oil image features, the quality of the transformer oil is analyzed to obtain transformer oil quality information. For example, a darker color of the oil may indicate that the oil has aged; a reduced transparency may indicate that there are impurities or moisture in the oil.

[0050] In operation S203, the internal structure image is corrected based on the quality information of the transformer oil to obtain a first image.

[0051] According to an embodiment of the present invention, the image parameters that need to be corrected are determined based on the quality information of the transformer oil. For example, if there are a lot of impurities in the oil that cause the image to be blurred, the sharpness of the image needs to be enhanced; if the color of the oil changes significantly, the color balance of the image needs to be adjusted.

[0052] According to an embodiment of the present invention, based on the image parameters to be corrected, an image processing algorithm can be used to correct the internal structure image. After the correction process, a first image is obtained. The image processing algorithm may include, for example, image enhancement, color correction, sharpening, etc. The first image can be used to reflect the actual condition inside the transformer, such as the condition of the transformer oil, whether there are defects inside the transformer, etc.

[0053] In operation S204, object detection is performed based on the first image to obtain a defect target detection result of the transformer.

[0054] According to an embodiment of the present invention, an object detection model can be used to perform feature extraction, classification and other processing on the first image to detect transformer defect targets in the first image and output defect target detection results, wherein the detection results may include location information, defect contours and morphological information of the transformer defect targets.

[0055] Based on this, the embodiment of the present invention can accurately evaluate the quality information of transformer oil by analyzing the transformer oil image features in the transformer internal structure image, thereby helping to timely discover the aging and pollution of the oil and other problems, and providing an important basis for the maintenance and management of the transformer. In addition, the object detection model is used to perform feature extraction and other processing on the corrected internal structure image to improve the detection accuracy of defective targets inside the transformer and better judge the operating status of the transformer.

[0056] According to an embodiment of the present invention, object detection is performed based on a first image to obtain a defect target detection result of a transformer, including: performing image enhancement processing on the first image based on adaptive histogram equalization and a self-calibrated illumination learning network to obtain an enhanced image; performing super-resolution reconstruction on the enhanced image to obtain a reconstructed image; and performing object detection and instance segmentation on the reconstructed image to obtain a defect target detection result of the transformer.

[0057] According to an embodiment of the present invention, in order to increase image contrast and highlight details, image enhancement processing can be performed on the first image based on adaptive histogram equalization, by calculating the local histogram of the image and then redistributing the brightness to change the image contrast, thereby retaining more image details.

[0058] According to an embodiment of the present invention, in order to adapt to the lighting conditions inside the transformer, after the first image is subjected to image enhancement processing based on adaptive histogram equalization, the contrast enhanced image can also be subjected to image enhancement processing based on a self-calibration illumination learning network. By constructing a weight-sharing illumination learning process that introduces a self-calibration module, the illumination change pattern and characteristics are learned, so that the transformer internal inspection equipment can accurately process and recognize images under various lighting conditions.

[0059] According to an embodiment of the present invention, an enhanced image obtained based on a self-calibrated illumination learning network and a contrast enhanced image obtained based on adaptive histogram equalization are multi-scale fused to obtain an enhanced image.

[0060] According to an embodiment of the present invention, the enhanced image is input into a super-resolution image enhancement network to perform super-resolution reconstruction on the enhanced image, so as to improve the image resolution while ensuring clarity, thereby obtaining a reconstructed image. The super-resolution image enhancement network may select a deep learning network with high reconstruction accuracy and generalization ability, such as a residual network or a generative adversarial network.

[0061] According to an embodiment of the present invention, the reconstructed image is input into a trained object detection model to perform object detection on the reconstructed image, so that the transformer and its potential defective areas in the reconstructed image can be identified.

[0062] According to an embodiment of the present invention, on the basis of object detection on the reconstructed image, instance segmentation can also be performed on the identified defect region information to extract the precise contour and morphological information of the transformer defect. For example, the defect region identified by the object detection model can be input into a trained instance segmentation model, and the precise segmentation mask of each defect region can be output to obtain the defect target detection result of the transformer, thereby determining the specific location and shape information of each defect based on the defect target detection result.

[0063] Based on this, an embodiment of the present invention utilizes adaptive histogram equalization and self-calibration illumination learning network fusion method to enhance the internal image of the transformer to improve the quality of the under-oil visual image inside the transformer, and utilizes an image super-resolution network to restore the lost details in the low-resolution internal image of the transformer, so that the overall system has a strong generalization ability to adapt to image segmentation under conditions such as illumination changes and pollution interference; in addition, object detection and instance segmentation are used to obtain more accurate defect contours and morphological information, which provides a basis for analyzing complex shapes such as electrical tree discharge defects, carbon mark defects, winding deformation and foreign body defects.

[0064] According to an embodiment of the present invention, image enhancement processing is performed on a first image based on adaptive histogram equalization and a self-calibrated illumination learning network to obtain an enhanced image, including: using a histogram equalization method to perform contrast enhancement on the first image to obtain a grayscale image; splicing the grayscale image with the first image according to the channel dimension to obtain a second image; performing multi-scale feature extraction on the second image to obtain a first feature map; inputting the first feature map into a self-calibrated illumination learning network to obtain a second feature map; performing multi-scale fusion on the second feature map and the grayscale image to obtain a third feature map; and using a residual network to process the third feature map to output an enhanced image.

[0065] According to an embodiment of the present invention, the contrast of the first image is enhanced by using a histogram equalization method to obtain a grayscale image. Specifically, the number of pixels of each grayscale value in the first image is counted, and the number of grayscale levels in the first image can be set to L, the grayscale range is from 0 to L-1, and the number of pixels in the first image is N. Then the cumulative distribution function (CDF) of the grayscale value k is expressed as follows:

[0066] (1);

[0067] In the formula, s k represents the cumulative distribution function of the gray value k; L represents the number of gray levels in the first image; N represents the number of pixels in the first image; n i Represents the number of pixels with grayscale value i.

[0068] Among them, the cumulative distribution function s of the gray value k kIt can be used to represent the mapping relationship from the original grayscale value of the first image to the grayscale value after equalization. Therefore, the cumulative distribution function s can be used k The original grayscale value is mapped to a new grayscale value, and the new grayscale value can be obtained by transforming the cumulative distribution function s k The value of is multiplied by the maximum gray level (L-1). At this time, the pixels with lower gray values ​​in the first image will be mapped to lower new gray values, and the pixels with higher gray values ​​will be mapped to higher new gray values, but the entire gray range will be used more evenly, so as to obtain a gray image with enhanced contrast.

[0069] Based on this, through histogram equalization, different brightness levels in the first image are more evenly distributed throughout the entire brightness range, so that areas where details were originally lost in dark or bright areas become visible, thereby enhancing the overall visual effect of the image.

[0070] According to an embodiment of the present invention, since the grayscale image only contains grayscale information but no color information, the grayscale image can be used as the fourth channel of the red, green and blue (RGB) image and spliced ​​with the first image according to the channel dimension to obtain a four-channel second image, wherein the four channels include a red channel, a green channel, a blue channel and a grayscale channel.

[0071] According to an embodiment of the present invention, the spliced ​​second image is input into the transformer internal image enhancement network, wherein the transformer internal image enhancement network may include multiple convolutional layers, pooling layers, fully connected layers, etc., to extract multi-scale features of the image and perform enhancement processing.

[0072] Specifically, multiple convolutional layers and pooling layers can be used to perform multi-scale feature extraction on the input four-channel second image, generate a high-level feature map, and use the high-level feature map as the first feature map. Among them, the convolutional layer can be used to capture local features in the image, while the pooling layer can be used to reduce the resolution of the feature map and extract the main features. By stacking multiple convolutional layers and pooling layers, the network can extract multi-scale features of the image, which contain both detail information and global information. The high-level feature map can include more abstract and advanced information in the image, such as the edges of the defect area, texture color and other features.

[0073] According to an embodiment of the present invention, the first feature map is input into a self-calibrated illumination learning network to output a second feature map. In low light conditions, the illumination component of the image is weak, resulting in a dark image as a whole and reduced contrast. Therefore, a self-calibrated illumination learning network based on Retinex theory can be used to separate the illumination component and the reflection component of the image, and enhance the image by optimizing the illumination component.

[0074] Specifically, based on the Retinex theory, the low-light observation image is the dot product of the clear image and the illumination, which is expressed as:

[0075] (2);

[0076] Where y represents the low-light observation image, x represents the illumination component, and z represents the desired clear image.

[0077] According to an embodiment of the present invention, in the low-light observation image model, illumination estimation can be regarded as the main optimization target. After obtaining accurate illumination, a clear image can be directly obtained according to the above relationship. In order to restore a clear image from low-light observation, a cascade illumination learning process with weight sharing can be adopted for the self-calibrated illumination learning network. The expression of the basic unit is as follows:

[0078] (3);

[0079] In the formula, Represents the basic unit, Indicates the current illumination. Indicates updated illumination, Represents the residual term of the tth stage, which is used to adjust the current illumination Predicted Update Illumination .

[0080] According to an embodiment of the present invention, in order to reduce the computational burden of the cascade model and achieve convergence between the results of each stage, a self-calibration module can be constructed for the self-calibration illumination learning network. The expression of the module is as follows:

[0081] (4);

[0082] In the formula, Represents the self-calibration module, represents the transformation input of each stage, represents a parameterized operator, Represents the clear image estimation. Among them, the transformation input of each stage It can be estimated based on the low-light observation image y and the clear image of the current stage Get. Parameterized operator For estimating from clear images Predict a correction amount to get the new transformation input.

[0083] According to an embodiment of the present invention, combining the basic unit of formula (3) and the self-calibration module of formula (4), the basic unit of the lighting optimization process can be restated as:

[0084] (5);

[0085] Specifically, first use the basic unit To update the current illumination , get the updated illumination , then use the self-calibration module To process the updated illuminance to get the new conversion input This process can be iterated until the output second feature map reaches the desired clear image quality.

[0086] Based on this, the self-calibrated illumination learning network combines the advantages of Retinex theory through a cascaded illumination learning method, which can effectively restore clear images from low-light observations. The basic unit is responsible for gradually optimizing the illumination, while the self-calibration module is responsible for achieving convergence of results between each stage, thereby reducing the computational cost.

[0087] According to an embodiment of the present invention, by combining unsupervised training loss, multi-scale fusion, and the use of convolutional layers and residual blocks, the learning ability of the self-calibrated illumination learning network can be further enhanced and the final generated enhanced image can be optimized.

[0088] Specifically, in order to enhance the generalization ability of the model in different scenarios, the unsupervised training loss is defined to consist of two parts: fidelity loss and smoothness loss. These two loss terms are combined through weighted coefficients to form a total loss function. Specifically, the expression of the total loss function is as follows:

[0089] (6);

[0090] In the formula, Represents the total loss function, L f represents the fidelity loss, L s represents the smoothing loss, and can all be expressed as weighted coefficients.

[0091] Among them, the fidelity loss L f It can be used to ensure that the enhanced image is consistent with the original low-light image in content. For example, this can be achieved by calculating some distance between the enhanced image and the original image (such as mean square error, structural similarity index, etc.). Smoothing loss L s It can be used to constrain the smoothness of the illumination component to avoid introducing unnecessary noise and artifacts during the enhancement process. For example, this can be achieved by calculating the gradient or Laplacian operator of the illumination component and penalizing it.

[0092] According to an embodiment of the present invention, after obtaining the preliminarily enhanced second feature map, in order to further improve the contrast and clarity of the image, the second feature map can be multi-scale fused with the contrast-enhanced grayscale image obtained by adaptive histogram equalization to obtain a third feature map.

[0093] For example, the preliminarily enhanced second feature map and the contrast-enhanced grayscale image can be downsampled and upsampled at different scales, and then fused at each scale. The fusion method can be a simple weighted average, maximum value selection, or a more complex fusion strategy. Through multi-scale fusion, the contrast and detail richness of the image can be further improved.

[0094] According to an embodiment of the present invention, after obtaining the fused third feature map, the fused third feature map can be further optimized using a convolutional layer and a residual network to obtain an enhanced image. The residual network can enhance these features and avoid the gradient vanishing problem through skip connections.

[0095] For example, the fused third feature map can be input into a residual network containing multiple convolutional layers and residual blocks, and the network parameters can be optimized through forward propagation and back propagation. During the training process, the unsupervised training loss defined above can be used as the optimization target to update the network weights through algorithms such as gradient descent.

[0096] According to an embodiment of the present invention, super-resolution reconstruction is performed on an enhanced image to obtain a reconstructed image, including: inputting the enhanced image into a generator of an adversarial network to perform super-resolution reconstruction on the enhanced image, and outputting the reconstructed image; wherein the adversarial network is trained by combining pixel-level loss, perceptual loss and adversarial loss.

[0097] According to an embodiment of the present invention, the enhanced image can be input into a generative adversarial network for super-resolution reconstruction, wherein the generative adversarial network (GAN) can be used to capture and reconstruct image details of the enhanced image while improving the resolution of the enhanced image.

[0098] Specifically, the generative adversarial network consists of two parts: a generator and a discriminator, wherein the generator can be used to generate realistic images, and the discriminator can be used to distinguish between the generated images and real images. Therefore, in one embodiment of the present invention, the enhanced image can be input into the generator of the generative adversarial network to map the input low-resolution image to the high-resolution image space, thereby performing super-resolution reconstruction, and outputting a reconstructed image.

[0099] According to an embodiment of the present invention, in order to enhance the robustness and adaptability of the generative adversarial network, the generative adversarial network can be trained by combining pixel-level loss, perceptual loss and adversarial loss respectively. The total loss function of the generative adversarial network is expressed as follows:

[0100] (7);

[0101] In the formula, L represents the total loss function of the generative adversarial network, L1 represents the pixel-level loss, represents the perceptual loss, L G represents adversarial loss, and can all be expressed as weighted coefficients.

[0102] According to an embodiment of the present invention, the pixel-level loss L1 can be used to measure the difference between the generated image and the real image at the pixel level. The pixel-level loss L1 can be implemented by calculating the mean square error (MSE) or mean absolute error (MAE) between the two. The pixel-level loss can ensure that the generated image is consistent with the real image in details.

[0103] According to an embodiment of the present invention, the perceived loss It can be used to measure the difference between the generated image and the real image in high-level features. This can be done with a pre-trained convolutional neural network (such as the VGG network), where the image is fed into the network and the difference between the feature maps is calculated. This ensures that the generated images are perceptually similar to real images.

[0104] According to an embodiment of the present invention, the adversarial loss L G It can be used to train the generator and discriminator in GAN. Among them, the adversarial loss L G It can ensure that the generated images are consistent with the real images in overall structure and style.

[0105] Based on this, through the generator of the adversarial network, the low-resolution details in the enhanced image can be mapped to the high-resolution space, thereby restoring clearer and more delicate image details; in addition, by combining multiple loss functions for training, the model can learn a richer image feature representation, combined with pixel-level loss, to ensure that the reconstructed image is highly consistent with the real high-resolution image at the pixel level, reducing image distortion and artifacts. The introduction of perceptual loss can further enhance the similarity of images in high-level features, making the reconstructed image closer to the real image in visual perception. This significantly improves the quality, naturalness and realism of the image, while enhancing the robustness and generalization ability of the model.

[0106] Figure 3A flow chart of an object detection and instance segmentation method according to an embodiment of the present invention is shown.

[0107] like Figure 3 As shown, the method includes operations S301 to S305.

[0108] In operation S301, the reconstructed image is input into an object detection network to obtain a target detection feature map.

[0109] According to an embodiment of the present invention, in order to enable the robot fish to accurately identify the defect type in the transformer oil, the object detection network can adopt a deep learning target detection network, such as the YOLO series model, among which the YOLOv10-s model can optimize the network structure and reduce the number of parameters to significantly improve the detection speed while maintaining a high detection accuracy.

[0110] According to a specific embodiment of the present invention, the reconstructed image is input into the YOLOv10-s model, and the forward propagation module in the YOLOv10-s model is used to extract target detection features from the reconstructed image to generate a target detection feature map containing information such as potential defect location, size, and confidence.

[0111] In operation S302, feature extraction is performed on the reconstructed image to obtain a fourth feature map.

[0112] According to an embodiment of the present invention, in order to make full use of the original image information, a convolutional neural network can also be used to extract low-level features of the reconstructed image to obtain a low-level feature map of the reconstructed image as the fourth feature map.

[0113] In operation S303, the target detection feature map and the fourth feature map are fused using an attention mechanism to obtain a cross feature map.

[0114] Specifically, a weight matrix may be calculated to weight the fourth feature map element-by-element multiplication, and then the weighted fourth feature map may be fused with the target detection feature map to obtain a cross feature map.

[0115] Among them, the weight matrix can be used to represent the correlation or importance between each position between the target detection feature map and the fourth feature map. This can be achieved by various methods, such as using additional convolutional layers to learn attention weights, or by calculating the similarity between the two feature maps (such as dot product, cosine similarity) to obtain weights. Fusion methods can include addition, element-by-element multiplication, splicing, etc.

[0116] In operation S304, the cross feature map is input into an instance segmentation network to obtain a pixel mask related to the detection target.

[0117] According to a specific embodiment of the present invention, the instance segmentation network may adopt an image segmentation model (SegmentAnything Model‌, SAM), which may simultaneously perform classification and segmentation tasks to improve segmentation accuracy and image segmentation efficiency.

[0118] According to an embodiment of the present invention, the fused cross feature map is input into the forward propagation module of SAM for instance segmentation. During the forward propagation process, the cross feature map passes through the convolution layer, pooling layer, etc. of the SAM network to generate a pixel mask related to the detection target, thereby achieving fine-grained segmentation of the object area.

[0119] According to an embodiment of the present invention, in order to achieve joint optimization of target detection and instance segmentation, after the cross feature map is connected to the forward propagation module of SAM, back propagation and optimization can also be performed through a joint loss function. The joint loss function may include the loss of the target detection part and the loss of the instance segmentation part.

[0120] According to a specific embodiment of the present invention, the loss function of the target detection part generally includes position loss, confidence loss and classification loss. Among them, the position loss (Location Loss) can be used to measure the position difference between the detection box and the real box. The confidence loss (Confidence Loss) can be used to measure the difference between the confidence of the detection box and the real confidence. The classification loss (Classification Loss) can be used to measure the classification accuracy of the detection box. Therefore, the loss function of the target detection part can be used to measure the position accuracy, confidence accuracy and classification accuracy of the detection box respectively. Specifically, the expression of the loss function of the target detection part is as follows:

[0121] (8);

[0122] In the formula, represents the loss function of the target detection part, represents the position loss, represents the confidence loss, represents the classification loss.

[0123] According to a specific embodiment of the present invention, the loss function of the instance segmentation part can adopt the cross-entropy loss (Cross-Entropy Loss) to measure the difference between the true mask and the predicted mask. Among them, the true mask (True Mask) can be used to represent the real target area. The predicted mask (Predicted Mask) can be used to represent the target area predicted by the network. Specifically, the expression of the loss function of the instance segmentation part is as follows:

[0124] (9);

[0125] In the formula, represents the loss function of the instance segmentation part, L CE represents the cross entropy loss, represents the prediction mask, Represents the true mask.

[0126] According to a specific embodiment of the present invention, the expression of the joint loss function is as follows:

[0127] (10);

[0128] Where, L u represents the joint loss function, and Represents the weight coefficient, which is used to balance the proportion of the two tasks of object detection and instance segmentation. These weight coefficients can be determined through experiments to obtain the best segmentation and detection performance.

[0129] According to a specific embodiment of the present invention, by calculating the gradient of the joint loss function, back propagation is performed to update the weight coefficient of the network. This process is iterated until the network converges or reaches a preset number of training rounds. After the training is completed, the cross feature map to be detected is input into the trained SAM network to obtain a pixel mask related to the detection target.

[0130] The pixel mask can be a two-dimensional array or image, and the value of each element or pixel indicates whether the pixel at the corresponding image position belongs to a transformer defect. These values ​​are usually binary (such as 0 or 1), or in some cases are probability values ​​(such as between 0 and 1).

[0131] In operation S305 , a defect target detection result of the transformer is obtained based on the pixel mask.

[0132] According to a specific embodiment of the present invention, if the pixel mask outputs a probability value, a preset threshold can be used to determine which pixels belong to the transformer defect area. For example, the preset threshold can be set to 0.5, and pixels greater than the threshold can be considered as part of the defect area.

[0133] According to a specific embodiment of the present invention, after determining which pixels belong to the transformer defect area, a connected region analysis may be performed on the binary pixel mask to find all interconnected pixel sets, which may be used to characterize different objects or defects in the image.

[0134] According to a specific embodiment of the present invention, for each connected area, its minimum circumscribed rectangle, i.e., a bounding box, can be calculated. This bounding box can be used to provide rough location and size information of the defect target. In addition to the bounding box, other defect features can also be extracted, such as area, perimeter, shape parameters (such as aspect ratio, roundness, etc.) and possible texture features.

[0135] According to a specific embodiment of the present invention, the detected defects can be further filtered and classified according to the extracted features to obtain the defect target detection result of the transformer. For example, an area threshold can be set to exclude too small noise areas, or a machine learning model can be used to classify defects. Among them, the defect target detection result of the transformer can include information such as the location, size and type of the defect.

[0136] Based on this, an embodiment of the present invention integrates target detection and instance segmentation. First, the forward propagation module of the object detection network is used to extract target detection features, which provides clues for the instance segmentation network and ensures that the forward propagation module of the instance segmentation network obtains the image features of target detection and instance segmentation. At the same time, a joint loss function including the two losses of target detection and instance segmentation is used to perform back propagation and optimization during the training process, so that the network can simultaneously learn how to accurately detect targets and finely segment instances, which is conducive to the network to better capture the contours and morphological information of complex shape defects such as electrical tree discharge defects, carbon mark defects, winding deformation and foreign body defects.

[0137] According to an embodiment of the present invention, based on the internal structure image, the quality information of the transformer oil inside the transformer is determined, including: extracting a first sub-image containing transformer components and a second sub-image containing background information from the internal structure image; determining the distortion information of the transformer oil based on the shape characteristics of the transformer components in the first sub-image and the actual shape characteristics of the transformer structure; performing abnormal pixel point detection on the second sub-image to determine at least one first abnormal pixel block, the first abnormal pixel block being composed of a plurality of abnormal pixel points; obtaining the color information of the transformer oil based on the pixel information of other pixel points in the second sub-image except for the at least one first abnormal pixel block; determining the impurity information of the transformer oil based on the number of abnormal pixel points contained in each of the at least one first abnormal pixel blocks, the impurity information being used to represent the size probability distribution of the abnormal pixel blocks.

[0138] According to an embodiment of the present invention, the quality information of transformer oil includes distortion information, color information and impurity information of transformer oil. Among them, the distortion information of transformer oil can be used to reflect abnormal changes in the physical or chemical properties of transformer oil, which may be caused by oxidation, thermal decomposition, pollution or chemical reaction with other substances. The color information of transformer oil can be used to reflect the aging degree and pollution degree of transformer oil. The impurity information of transformer oil can be used to reflect the content of solid particles, moisture, gas or other pollutants that may exist in transformer oil.

[0139] According to an embodiment of the present invention, a first sub-image and a second sub-image are respectively extracted from an internal structure image of a transformer, wherein the first sub-image can be used to represent a foreground image including main components of the transformer (such as a coil, an iron core, etc.), and the second sub-image can be used to represent a background image including background information, wherein the background information may include quality information of transformer oil.

[0140] According to an embodiment of the present invention, for the first sub-image, shape features of the transformer components in the first sub-image can be extracted, and the extracted shape features can be compared with known actual shape features of the transformer components. Based on the comparison results, distortion information of the transformer oil can be determined, such as whether there is deterioration caused by oil level changes, leakage, etc.

[0141] According to an embodiment of the present invention, for the second sub-image, each pixel point of the second sub-image is traversed, and abnormal pixel points are detected based on a set abnormality determination standard (such as color, brightness, etc.) to determine at least one first abnormal pixel block. The first abnormal pixel block is composed of multiple abnormal pixel points. For example, adjacent abnormal pixel points can be combined into at least one first abnormal pixel block.

[0142] According to an embodiment of the present invention, for the second sub-image, pixel information of the remaining pixels except the first abnormal pixel block, such as color information (such as RGB values) of the remaining pixels is analyzed, and color information of the transformer oil can be obtained based on the analysis result.

[0143] According to an embodiment of the present invention, the number of abnormal pixels in each first abnormal pixel block is counted, and the impurity information of the transformer oil is determined based on the number of abnormal pixels. For example, when the impurity distribution in the transformer oil is uniform, the impurity probability distribution of the transformer oil can be determined based on the size probability distribution of the abnormal pixel blocks.

[0144] According to an embodiment of the present invention, based on the quality information of transformer oil, an internal structure image is corrected to obtain a first image, including: performing abnormal pixel point detection on the internal structure image to determine a plurality of second abnormal pixel blocks; determining the confidence of each of the plurality of second abnormal pixel blocks based on the number of abnormal pixel points contained in each of the plurality of second abnormal pixel blocks and the size probability distribution of the abnormal pixel blocks represented by impurity information; determining at least one target abnormal pixel block from the plurality of second abnormal pixel blocks based on a confidence threshold; eliminating at least one target abnormal pixel block in the internal structure image to obtain a third image; performing color correction on the third image based on color information to obtain a fourth image; and correcting the fourth image based on distortion information to obtain the first image.

[0145] According to an embodiment of the present invention, for the internal structure image of the transformer, each pixel point of the internal structure image is traversed, and abnormal pixel points on the internal structure image are detected based on a set abnormality determination standard (such as color, brightness, etc.) to determine a plurality of second abnormal pixel blocks. The second abnormal pixel block is composed of a plurality of abnormal pixel points on the internal structure image. For example, adjacent abnormal pixel points on the internal structure image can be combined into a plurality of second abnormal pixel blocks.

[0146] According to an embodiment of the present invention, based on the number of abnormal pixels in each second abnormal pixel block and known impurity information (i.e., the size probability distribution of the first abnormal pixel block), the confidence of each second abnormal pixel block is calculated, wherein the confidence can be used to reflect the probability that the second abnormal pixel block is a real impurity.

[0147] According to the embodiment of the present invention, by setting a confidence threshold, at least one target abnormal pixel block with a confidence higher than the threshold can be screened out from the plurality of second abnormal pixel blocks.

[0148] According to an embodiment of the present invention, at least one target abnormal pixel block in the internal structure image is eliminated to obtain a processed third image, thereby improving the accuracy of subsequent correction by removing noise and interference in the image.

[0149] According to an embodiment of the present invention, the third image is color corrected based on the known color information of the transformer oil, thereby obtaining the fourth image. Since the color of the photo actually taken is a combination of the actual color and the color of the oil, the actual color can be deduced from the color of the photo actually taken and the known color of the oil through color correction.

[0150] According to an embodiment of the present invention, based on the known distortion information of transformer oil, the fourth image after color correction can be further subjected to distortion correction to obtain the first image. As the transformer oil changes with its texture, the degree of light deflection will change, thereby causing image distortion. Therefore, the image distortion caused by light deflection can be eliminated through distortion correction.

[0151] According to the embodiment of the present invention, after the above-mentioned color correction and further correction, the first image finally obtained can be a more accurate, clear, noise-free and distortion-free internal structure image. This image not only reflects the true shape and position of the internal components of the transformer, but also eliminates the color deviation and image distortion caused by the transformer oil. Through the corrected image, the status and performance of the internal components of the transformer can be analyzed more accurately, potential safety hazards and fault points can be discovered in time, and strong support can be provided for the maintenance and repair of the transformer.

[0152] According to an embodiment of the present invention, the defect target detection method for robot fish operation inside a transformer also includes: performing data enhancement of samples based on multiple defect types of the transformer to obtain multiple training samples; and using the multiple training samples to train a self-calibrated illumination learning network, an adversarial network, an object detection network, and an instance segmentation network.

[0153] According to an embodiment of the present invention, data enhancement of an initial sample is performed based on multiple defect types of a transformer to obtain multiple training samples, including: for each defect type, extracting a first defect target image corresponding to the defect type from a defect library; randomly scaling the first defect target image to obtain a second defect target image; and superimposing the second defect target image on the initial sample to obtain a training sample corresponding to the defect type.

[0154] According to an embodiment of the present invention, internal defects of the transformer may include transformer body defects and foreign body defects. Among them, transformer body defects may include electrical tree discharge defects, carbon mark defects and winding deformation defects. Transformer foreign body defects may include metal gaskets, metal nuts, metal bolts, rubber washers, rubber sheets and crepe paper, etc.

[0155] According to an embodiment of the present invention, in order to construct the data set required for the training model, it is necessary to collect enough samples for each defect type. However, due to the scarcity of real transformer internal defect images, it is difficult to directly obtain a large number of high-quality defect images. Therefore, the data set can be expanded based on the data enhancement method.

[0156] Specifically, for transformer body defects, a first defect target image corresponding to a specific defect type can be extracted from the body defects, and the extracted first defect target image can be randomly scaled to obtain a second defect target image to simulate the defect size changes observed at different distances and viewing angles. The scaled second defect target image is then superimposed on multiple defect-free transformer internal images to generate training samples corresponding to a specific defect type.

[0157] Specifically, for transformer foreign body defects, a mask image of a specific foreign body can be extracted from the foreign body defect library, and the foreign body mask can be randomly scaled to adapt to different sizes and positions. The scaled foreign body mask is then superimposed on the transformer internal image to simulate the different positions and sizes of foreign bodies inside the transformer.

[0158] According to an embodiment of the present invention, Gaussian blur processing is applied to the training samples after data enhancement to improve their visual consistency, making them closer to real defect images, helping to reduce noise and irregularities in the images and improve the robustness of the model. At the same time, the enhanced training samples are used to train the self-calibrated illumination learning network, the adversarial network, the object detection network and the instance segmentation network to improve the detection and segmentation accuracy of internal defects of the transformer.

[0159] Figure 4 A block diagram of a defect target detection device for robot fish operation inside a transformer according to an embodiment of the present invention is shown.

[0160] like Figure 4 As shown, the device includes an identity recognition signal acquisition module 410 , a determination module 420 , a correction module 430 and a detection module 440 .

[0161] The acquisition module 410 is used to obtain the internal structure image of the transformer collected by the robot fish inside the transformer in response to the transformer internal state detection request.

[0162] The determination module 420 is used to determine the quality information of the transformer oil inside the transformer based on the internal structure image.

[0163] The correction module 430 is used to correct the internal structure image based on the quality information of the transformer oil to obtain a first image.

[0164] The detection module 440 is used to perform object detection based on the first image to obtain a defect target detection result of the transformer.

[0165] According to an embodiment of the present invention, the detection module 440 includes an image enhancement submodule, a reconstruction submodule, and a detection and segmentation submodule.

[0166] The image enhancement submodule is used to perform image enhancement processing on the first image based on adaptive histogram equalization and a self-calibrated illumination learning network to obtain an enhanced image.

[0167] The reconstruction submodule is used to perform super-resolution reconstruction on the enhanced image to obtain a reconstructed image.

[0168] The detection and segmentation submodule is used to perform object detection and instance segmentation on the reconstructed image to obtain the defect target detection result of the transformer.

[0169] According to an embodiment of the present invention, the image enhancement submodule includes an enhancement unit, a splicing unit, a first feature extraction unit, a first processing unit, a first fusion unit, and a second processing unit.

[0170] The enhancement unit is used to perform contrast enhancement on the first image by using a histogram equalization method to obtain a grayscale image.

[0171] The splicing unit is used to splice the grayscale image with the first image according to the channel dimension to obtain a second image.

[0172] The first feature extraction unit is used to perform multi-scale feature extraction on the second image to obtain a first feature map.

[0173] The first processing unit is used to input the first feature map into the self-calibration illumination learning network to obtain a second feature map.

[0174] The first fusion unit is used to perform multi-scale fusion of the second feature map and the grayscale image to obtain a third feature map.

[0175] The second processing unit is used to process the third feature map by using a residual network, and output an enhanced image.

[0176] According to an embodiment of the present invention, the reconstruction submodule includes a third processing unit.

[0177] The third processing unit is used to input the enhanced image into the generator of the adversarial network to perform super-resolution reconstruction on the enhanced image and output a reconstructed image.

[0178] According to an embodiment of the present invention, the detection and segmentation submodule includes a fourth processing unit, a second feature extraction unit, a second fusion unit, a fifth processing unit, and a first determination unit.

[0179] The fourth processing unit is used to input the reconstructed image into the object detection network to obtain a target detection feature map.

[0180] The second feature extraction unit is used to extract features from the reconstructed image to obtain a fourth feature map.

[0181] The second fusion unit is used to fuse the target detection feature map and the fourth feature map by using the attention mechanism to obtain a cross feature map.

[0182] The fifth processing unit is used to input the cross feature map into the instance segmentation network to obtain a pixel mask related to the detection target.

[0183] The first determination unit is used to obtain a defect target detection result of the transformer based on the pixel mask.

[0184] According to an embodiment of the present invention, the determination module 420 includes a first extraction submodule, a first determination submodule, a first detection submodule, a second determination submodule, and a third determination submodule.

[0185] The first extraction submodule is used to extract a first sub-image containing transformer components and a second sub-image containing background information from the internal structure image.

[0186] The first determination submodule is used to determine the distortion information of the transformer oil based on the shape features of the transformer components in the first sub-image and the actual shape features of the transformer structure.

[0187] The first detection submodule is used to perform abnormal pixel point detection on the second sub-image to determine at least one first abnormal pixel block, where the first abnormal pixel block is composed of a plurality of abnormal pixel points.

[0188] The second determination submodule is used to obtain color information of the transformer oil based on pixel information of other pixel points in the second sub-image except for the at least one first abnormal pixel block.

[0189] The third determination submodule is used to determine the impurity information of the transformer oil based on the number of abnormal pixel points contained in each of the at least one first abnormal pixel blocks, where the impurity information is used to represent the size probability distribution of the abnormal pixel blocks.

[0190] According to an embodiment of the present invention, the correction module 430 includes a second detection submodule, a fourth determination submodule, a fifth determination submodule, a rejection processing submodule, a color correction submodule, and an image correction submodule.

[0191] The second detection submodule is used to perform abnormal pixel point detection on the internal structure image to determine a plurality of second abnormal pixel blocks.

[0192] The fourth determination submodule is used to determine the confidence of each of the plurality of second abnormal pixel blocks based on the number of abnormal pixel points each of the plurality of second abnormal pixel blocks contains and the size probability distribution of the abnormal pixel blocks represented by the impurity information.

[0193] The fifth determination submodule is configured to determine at least one target abnormal pixel block from the plurality of second abnormal pixel blocks based on a confidence threshold.

[0194] The elimination processing submodule is used to eliminate at least one target abnormal pixel block in the internal structure image to obtain a third image.

[0195] The color correction submodule is used to perform color correction on the third image based on the color information to obtain a fourth image.

[0196] The image correction submodule is used to correct the fourth image based on the distortion information to obtain the first image.

[0197] According to an embodiment of the present invention, a defect target detection device for robot fish operation inside a transformer includes a data enhancement module and a training module.

[0198] The data enhancement module is used to perform data enhancement on samples based on multiple defect types of the transformer to obtain multiple training samples.

[0199] A training module for training the self-calibrated illumination learning network, adversarial network, object detection network, and instance segmentation network using multiple training samples.

[0200] According to an embodiment of the present invention, the data enhancement module includes a second extraction submodule and a scaling submodule.

[0201] The second extraction submodule is used to extract, for each defect type, a first defect target image corresponding to the defect type from the defect library.

[0202] The scaling submodule is used to randomly scale the first defect target image to obtain a second defect target image. The second defect target image is superimposed on the initial sample to obtain a training sample corresponding to the defect type.

[0203] According to the embodiments of the present invention, any one or more of the modules, submodules, units, and subunits, or at least part of the functions of any one of them can be implemented in one module. According to the embodiments of the present invention, any one or more of the modules, submodules, units, and subunits can be split into multiple modules for implementation. According to the embodiments of the present invention, any one or more of the modules, submodules, units, and subunits can be at least partially implemented as hardware circuits, such as field programmable gate arrays (FPGAs), programmable logic arrays (PLAs), systems on chips, systems on substrates, systems on packages, application specific integrated circuits (ASICs), or can be implemented by hardware or firmware in any other reasonable way of integrating or packaging the circuit, or by any one of the three implementation methods of software, hardware, and firmware, or by a proper combination of any of them. Alternatively, according to the embodiments of the present invention, one or more of the modules, submodules, units, and subunits can be at least partially implemented as computer program modules, and when the computer program modules are run, the corresponding functions can be executed.

[0204] For example, any multiple of the acquisition module 410, the determination module 420, the correction module 430, and the detection module 440 can be combined in one module / unit / sub-unit for implementation, or any one of the modules / units / sub-units can be split into multiple modules / units / sub-units. Alternatively, at least part of the functions of one or more of these modules / units / sub-units can be combined with at least part of the functions of other modules / units / sub-units and implemented in one module / unit / sub-unit. According to an embodiment of the present invention, at least one of the acquisition module 410, the determination module 420, the correction module 430, and the detection module 440 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or by any one of the three implementation methods of software, hardware, and firmware, or by a suitable combination of any of them. Alternatively, at least one of the acquisition module 410 , the determination module 420 , the correction module 430 , and the detection module 440 may be at least partially implemented as a computer program module, and when the computer program module is executed, a corresponding function may be performed.

[0205] It should be noted that the defect target detection device part in the embodiment of the present invention corresponds to the defect target detection method part in the embodiment of the present invention. The description of the defect target detection device part specifically refers to the defect target detection method part, which will not be repeated here.

[0206] Figure 5A block diagram of an electronic device suitable for implementing a defect target detection method for a robot fish operation inside a transformer according to an embodiment of the present invention is shown. Figure 5 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0207] like Figure 5 As shown, the electronic device according to an embodiment of the present invention includes a processor 501, which can perform various appropriate actions and processes according to the program stored in the read-only memory ROM 502 or the program loaded from the storage part 508 to the random access memory RAM 503. The processor 501 may include, for example, a general-purpose microprocessor (such as a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (for example, an application-specific integrated circuit (ASIC)), etc. The processor 501 may also include an on-board memory for caching purposes. The processor 501 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present invention.

[0208] In RAM 503, various programs and data required for the operation of the electronic device are stored. The processor 501, ROM 502 and RAM 503 are connected to each other via a bus 504. The processor 501 performs various operations of the method flow according to the embodiment of the present invention by executing the programs in ROM 502 and / or RAM 503. It should be noted that the program can also be stored in one or more memories other than ROM 502 and RAM 503. The processor 501 can also perform various operations of the method flow according to the embodiment of the present invention by executing the programs stored in the one or more memories.

[0209] According to an embodiment of the present invention, the electronic device may further include an input / output (I / O) interface 505, which is also connected to the bus 504. The electronic device may further include one or more of the following components connected to the input / output (I / O) interface 505: an input portion 506 including a keyboard, a mouse, etc.; an output portion 507 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 508 including a hard disk, etc.; and a communication portion 509 including a network interface card such as a LAN card, a modem, etc. The communication portion 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to the input / output (I / O) interface 505 as needed. A removable medium 511, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 510 as needed, so that a computer program read therefrom is installed into the storage portion 508 as needed.

[0210] According to an embodiment of the present invention, the method flow according to an embodiment of the present invention can be implemented as a computer software program. For example, an embodiment of the present invention includes a computer program product, which includes a computer program carried on a computer-readable storage medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 509, and / or installed from the removable medium 511. When the computer program is executed by the processor 501, the above-mentioned functions defined in the system of the embodiment of the present invention are executed. According to an embodiment of the present invention, the system, equipment, device, module, unit, etc. described above can be implemented by a computer program module.

[0211] The present invention also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiment; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present invention is implemented.

[0212] According to an embodiment of the present invention, the computer-readable storage medium may be a non-volatile computer-readable storage medium. For example, it may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, an apparatus or a device.

[0213] For example, according to an embodiment of the present invention, the computer-readable storage medium may include the ROM 502 and / or the RAM 503 described above and / or one or more memories other than the ROM 502 and the RAM 503 .

[0214] An embodiment of the present invention also includes a computer program product, which includes a computer program, which contains program code for executing the method provided by the embodiment of the present invention. When the computer program product runs on an electronic device, the program code is used to enable the electronic device to implement the defect target detection method for the robot fish operation inside the transformer provided by the embodiment of the present invention.

[0215] When the computer program is executed by the processor 501, the above functions defined in the system / device of the embodiment of the present invention are executed. According to the embodiment of the present invention, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0216] In one embodiment, the computer program may rely on tangible storage media such as optical storage devices, magnetic storage devices, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of signals on a network medium, and downloaded and installed through the communication part 509, and / or installed from the removable medium 511. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0217] According to an embodiment of the present invention, the program code for executing the computer program provided by the embodiment of the present invention can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (e.g., using an Internet service provider to connect through the Internet).

[0218] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present invention. In this regard, each box in the flowchart or block diagram may represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box may also occur in an order different from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions. It can be understood by those skilled in the art that the features recorded in the various embodiments of the present invention can be combined and / or combined in various ways, even if such a combination or combination is not explicitly recorded in the present invention. In particular, without departing from the spirit and teachings of the present invention, the features recorded in the various embodiments of the present invention can be combined and / or combined in various ways. All such combinations and / or combinations fall within the scope of the present invention.

[0219] The embodiments of the present invention are described above. However, these embodiments are only for the purpose of illustration, and are not intended to limit the scope of the present invention. Although each embodiment is described above, it does not mean that the measures in each embodiment cannot be used in combination advantageously. Without departing from the scope of the present invention, those skilled in the art may make various substitutions and modifications, which should all fall within the scope of the present invention.

Claims

1. A defect target detection method for robot fish operation inside a transformer, characterized in that: The defect target detection method comprises: In response to a transformer internal state detection request, acquiring an internal structure image of the transformer collected by a robot fish inside the transformer; Based on the internal structure image, determining quality information of transformer oil inside the transformer; Based on the quality information of the transformer oil, the internal structure image is corrected to obtain a first image; Performing object detection based on the first image to obtain a defect target detection result of the transformer; The quality information of the transformer oil includes distortion information, color information and impurity information of the transformer oil; Wherein, determining the quality information of the transformer oil inside the transformer based on the internal structure image includes: Extracting a first sub-image containing transformer components and a second sub-image containing background information from the internal structure image; Determining the distortion information of the transformer oil based on the shape features of the transformer components in the first sub-image and the actual shape features of the transformer structure; Performing abnormal pixel point detection on the second sub-image to determine at least one first abnormal pixel block, where the first abnormal pixel block is composed of a plurality of abnormal pixel points; Based on pixel information of other pixel points in the second sub-image except the at least one first abnormal pixel block, obtaining color information of the transformer oil; Based on the number of abnormal pixel points respectively included in the at least one first abnormal pixel block, determining the impurity information of the transformer oil, wherein the impurity information is used to represent the size probability distribution of the abnormal pixel block; The step of correcting the internal structure image based on the quality information of the transformer oil to obtain a first image includes: Performing abnormal pixel detection on the internal structure image to determine a plurality of second abnormal pixel blocks; Determine the confidence level of each of the plurality of second abnormal pixel blocks based on the number of abnormal pixel points each of the plurality of second abnormal pixel blocks contains and the size probability distribution of the abnormal pixel blocks represented by the impurity information; Based on a confidence threshold, determining at least one target abnormal pixel block from the plurality of second abnormal pixel blocks; Eliminate at least one target abnormal pixel block in the internal structure image to obtain a third image; Performing color correction on the third image based on the color information to obtain a fourth image; The fourth image is corrected based on the distortion information to obtain the first image.

2. The defect target detection method according to claim 1, characterized in that: The performing object detection based on the first image to obtain a defect target detection result of the transformer includes: Performing image enhancement processing on the first image based on adaptive histogram equalization and a self-calibrated illumination learning network to obtain an enhanced image; Performing super-resolution reconstruction on the enhanced image to obtain a reconstructed image; Object detection and instance segmentation are performed on the reconstructed image to obtain a defect target detection result of the transformer.

3. The defect target detection method according to claim 2, characterized in that: The performing image enhancement processing on the first image based on the adaptive histogram equalization and self-calibration illumination learning network to obtain an enhanced image includes: Performing contrast enhancement on the first image using a histogram equalization method to obtain a grayscale image; Concatenate the grayscale image and the first image according to the channel dimension to obtain a second image; Performing multi-scale feature extraction on the second image to obtain a first feature map; Inputting the first feature map into a self-calibrated illumination learning network to obtain a second feature map; Perform multi-scale fusion of the second feature map and the grayscale image to obtain a third feature map; The third feature map is processed using a residual network, and the enhanced image is outputted.

4. The defect target detection method according to claim 2, characterized in that: The super-resolution reconstruction of the enhanced image to obtain a reconstructed image includes: Inputting the enhanced image into a generator of an adversarial network to perform super-resolution reconstruction on the enhanced image to obtain the reconstructed image; The adversarial network is trained by combining pixel-level loss, perceptual loss and adversarial loss.

5. The defect target detection method according to claim 2, characterized in that: The performing object detection and instance segmentation on the reconstructed image to obtain a defect target detection result of the transformer includes: Inputting the reconstructed image into an object detection network to obtain a target detection feature map; Performing feature extraction on the reconstructed image to obtain a fourth feature map; Using an attention mechanism, the target detection feature map and the fourth feature map are fused to obtain a cross feature map; Inputting the cross feature map into an instance segmentation network to obtain a pixel mask related to the detection target; Based on the pixel mask, a defect target detection result of the transformer is obtained.

6. The defect target detection method according to any one of claims 1 to 5, characterized in that: The defect target detection method also includes: Performing data enhancement on initial samples based on multiple defect types of the transformer to obtain multiple training samples; The plurality of training samples are used to train a self-calibrated illumination learning network, an adversarial network, an object detection network, and an instance segmentation network.

7. The defect target detection method according to claim 6, characterized in that: The data enhancement of the initial sample based on the multiple defect types of the transformer to obtain multiple training samples includes: For each of the defect types, extracting a first defect target image corresponding to the defect type from a defect library; Randomly scaling the first defect target image to obtain a second defect target image; The second defect target image is superimposed on the initial sample to obtain a training sample corresponding to the defect type.

8. A defect target detection device for robot fish operation inside a transformer, comprising: An acquisition module, configured to acquire, in response to a transformer internal state detection request, an internal structure image of the transformer collected by a robot fish inside the transformer; A determination module, configured to determine quality information of transformer oil inside the transformer based on the internal structure image; A correction module, used for correcting the internal structure image based on the quality information of the transformer oil to obtain a first image; A detection module, configured to perform object detection based on the first image to obtain a defect target detection result of the transformer; The quality information of the transformer oil includes distortion information, color information and impurity information of the transformer oil; Wherein, the determination module includes: A first extraction submodule, configured to extract a first sub-image containing transformer components and a second sub-image containing background information from the internal structure image; A first determination submodule, configured to determine the distortion information of the transformer oil based on the shape features of the transformer components in the first sub-image and the actual shape features of the transformer structure; A first detection submodule, configured to perform abnormal pixel point detection on the second sub-image to determine at least one first abnormal pixel block, where the first abnormal pixel block is composed of a plurality of abnormal pixel points; A second determination submodule, configured to obtain color information of the transformer oil based on pixel information of other pixel points in the second sub-image except for the at least one first abnormal pixel block; A third determination submodule, configured to determine impurity information of the transformer oil based on the number of abnormal pixel points contained in each of the at least one first abnormal pixel blocks, wherein the impurity information is used to represent a size probability distribution of the abnormal pixel blocks; Wherein, the correction module comprises: A second detection submodule, configured to perform abnormal pixel point detection on the internal structure image to determine a plurality of second abnormal pixel blocks; A fourth determination submodule, configured to determine the confidence level of each of the plurality of second abnormal pixel blocks based on the number of abnormal pixel points each of the plurality of second abnormal pixel blocks contains and the size probability distribution of the abnormal pixel blocks represented by the impurity information; a fifth determination submodule, configured to determine at least one target abnormal pixel block from the plurality of second abnormal pixel blocks based on a confidence threshold; A removal processing submodule, used for removing at least one target abnormal pixel block in the internal structure image to obtain a third image; a color correction submodule, configured to perform color correction on the third image based on the color information to obtain a fourth image; An image correction submodule is used to correct the fourth image based on the distortion information to obtain the first image.

Images