Transformer internal health state monitoring method and device based on internal inspection robot

Abstract

The invention provides a transformer internal health state monitoring method and device based on an internal inspection robot, and mainly relates to the technical field of image processing. The transformer internal health state monitoring method comprises the following steps: in response to a transformer health state monitoring request, obtaining a plurality of initial images collected by an internal inspection robot of a transformer; performing feature matching based on the plurality of initial images to obtain a key point matching result among the plurality of initial images; performing depth alignment splicing on the plurality of initial images based on a key point matching result among the plurality of initial images to obtain a first spliced image; performing rectangular correction on the first spliced image to obtain a second spliced image; and performing health state detection based on the second spliced image to obtain an internal health state monitoring result of the transformer.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and more specifically, to a method and device for monitoring the internal health status of a transformer based on an internal inspection robot. Background Art

[0002] With the rapid development of the power industry, transformers are vital equipment in power systems, and the monitoring and maintenance of their operating status are particularly important. In recent years, with the continuous advancement of robotics technology, robots have been increasingly used in transformer internal inspection.

[0003] However, due to the lighting conditions, oil pollution in transformer oil and the complexity of the equipment itself, the feature points are sparse and highly repetitive. Traditional feature extraction algorithms find it difficult to accurately extract enough feature points, which greatly reduces the accuracy of image matching and alignment. In addition, the transformer has a complex multi-layer structure inside, and there are obvious depth differences between images at different levels. Existing image stitching technologies are mostly based on a single homography matrix for image alignment, which cannot effectively handle this depth difference, which causes the stitched images to often have misalignment and ghosting phenomena. Summary of the invention

[0004] In view of this, the present invention provides a method and device for monitoring the internal health status of a transformer based on an internal inspection robot.

[0005] One aspect of the present invention provides a method for monitoring the internal health status of a transformer based on an internal inspection robot, comprising: in response to a transformer health status monitoring request, obtaining multiple initial images collected by the internal inspection robot of the transformer; performing feature matching based on the above multiple initial images to obtain key point matching results between the above multiple initial images; performing depth alignment and stitching on the above multiple initial images based on the key point matching results between the above multiple initial images to obtain a first stitched image; performing rectangularization correction on the above first stitched image to obtain a second stitched image; performing health status detection based on the above second stitched image to obtain the internal health status monitoring result of the above transformer.

[0006] Another aspect of the present invention provides a transformer internal health status monitoring device based on an internal inspection robot, including: an acquisition module, used to respond to a transformer health status monitoring request and acquire multiple initial images collected by the transformer internal inspection robot; a matching module, used to perform feature matching based on the above multiple initial images to obtain key point matching results between the above multiple initial images; a stitching module, used to perform depth alignment and stitching on the above multiple initial images based on the key point matching results between the above multiple initial images to obtain a first stitched image; a correction module, used to perform rectangularization correction on the above first stitched image to obtain a second stitched image; a detection module, used to perform health status detection based on the above second stitched image to obtain the internal health status monitoring result of the above transformer.

[0007] Another aspect of the present invention provides an electronic device, comprising: one or more processors; and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method as described above.

[0008] Another aspect of the present invention provides a computer-readable storage medium storing computer-executable instructions, which are used to implement the above method when executed.

[0009] Another aspect of the present invention provides a computer program product, the computer program product comprising computer executable instructions, the instructions being used to implement the method as described above when being executed.

[0010] According to the embodiments of the present invention, feature matching and depth alignment methods are used to accurately capture key feature points in images, and accurate alignment between images is achieved based on these key feature points. This method not only takes into account the overall transformation of the image, but also takes into account the adjustment of local details, thereby reducing ghosting and blurring problems. In addition, by further correcting the image boundaries and details to naturally process the transition of the image boundaries, the spliced ​​image is smoother and more natural at the boundaries, thereby providing strong data support for transformer health status monitoring. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The above and other objects, features and advantages of the present invention will become more apparent through the following description of the embodiments of the present invention with reference to the accompanying drawings.

[0012] Figure 1 An exemplary system architecture is shown to which a transformer internal health status monitoring method based on an internal inspection robot can be applied according to an embodiment of the present invention.

[0013] Figure 2A flow chart of a defect target detection method for a robot fish operation inside a transformer according to an embodiment of the present invention is shown.

[0014] Figure 3 A flow chart of a method for determining key point matching results between multiple initial images according to an embodiment of the present invention is shown.

[0015] Figure 4 A schematic diagram of the internal image of a transformer after processing according to a specific embodiment of the present invention is shown.

[0016] Figure 5 A block diagram of a transformer internal health status monitoring device based on an internal inspection robot according to an embodiment of the present invention is shown.

[0017] Figure 6 A block diagram of an electronic device suitable for implementing a transformer internal health status monitoring method based on an internal inspection robot according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0018] Below, embodiments of the present invention will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present invention. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of embodiments of the present invention. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of concepts of the present invention.

[0019] The terms used herein are only for describing specific embodiments and are not intended to limit the present invention. The terms "comprise", "include", etc. used herein indicate the existence of the features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0020] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0021] When using expressions such as "at least one of A, B, and C, etc.", they should generally be interpreted according to the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0022] With the rapid development of the power industry, transformers are vital equipment in power systems, and the monitoring and maintenance of their operating status are particularly important. In recent years, with the continuous advancement of robotics technology, robots have been increasingly used in transformer internal inspection.

[0023] The application of inspection robots in transformer internal inspection mainly relies on their high-precision sensors and image acquisition equipment. These inspection robots can penetrate deep into the transformer oil and make detailed observations and records of each component inside the transformer. Through the collected image data, technicians can analyze the transformer's insulation condition, winding deformation, oil level, and potential problems such as partial discharge. This inspection method not only improves the accuracy and efficiency of inspection, but also greatly reduces the risk and cost of manual inspection.

[0024] Although the inspection robot has achieved remarkable results in transformer internal inspection, the problem of image stitching technology still exists. Due to the lighting conditions, oil pollution and the complexity of the equipment itself in the transformer oil, the feature points are sparse and repetitive. It is difficult for traditional feature extraction algorithms to accurately extract enough feature points, resulting in a significant reduction in the accuracy of image matching and alignment. In addition, the transformer has a complex multi-layer structure, and there are obvious depth differences between images at different levels. The existing image stitching technology is mostly based on a single homography matrix for image alignment, which cannot effectively handle this depth difference, resulting in the stitched image often being misaligned and ghosting. These problems not only affect the accuracy and reliability of image stitching, but also limit the inspection robot's comprehensive assessment of the internal state of the transformer.

[0025] Based on this, an embodiment of the present invention provides a method for monitoring the internal health status of a transformer based on an internal inspection robot, including: in response to a transformer health status monitoring request, obtaining multiple initial images collected by the internal inspection robot of the transformer; performing feature matching based on the multiple initial images to obtain key point matching results between the multiple initial images; performing depth alignment and stitching on the multiple initial images based on the key point matching results between the multiple initial images to obtain a first stitched image; performing rectangularization correction on the first stitched image to obtain a second stitched image; performing health status detection based on the second stitched image to obtain an internal health status monitoring result of the transformer.

[0026] Figure 1 The exemplary system architecture of the transformer internal health status monitoring method based on the internal inspection robot according to the embodiment of the present invention is shown. It should be noted that: Figure 1 The examples shown are merely examples of system architectures to which the embodiments of the present invention may be applied, to help those skilled in the art understand the technical content of the present invention, but do not mean that the embodiments of the present invention may not be used in other devices, systems, environments or scenarios.

[0027] like Figure 1 As shown, the system architecture according to this embodiment may include a transformer 1, a transformer internal inspection device 2, and a processing device 3. A medium such as a communication link such as a network is provided between the transformer internal inspection device 2 and the processing device 3. The network may include various connection types, such as wired and / or wireless communication links, etc.

[0028] The transformer internal inspection device 2 may be located inside the transformer 1, and may interact with the processing device 3 through a network to receive or send information, etc. For example, the transformer internal inspection device 2 may be an internal inspection robot or other device.

[0029] The processing device 3 can be set at any position outside the transformer 1. The processing device 3 can be a server that provides various services to analyze and process the data such as the internal structure image of the transformer collected by the internal inspection robot inside the transformer.

[0030] It should be noted that the transformer internal health status monitoring method based on the internal inspection robot provided in the embodiment of the present invention can generally be executed by the processing device 3. Correspondingly, the transformer internal health status monitoring device based on the internal inspection robot provided in the embodiment of the present invention can generally be arranged in the processing device 3. The transformer internal health status monitoring method based on the internal inspection robot provided in the embodiment of the present invention can also be executed by a server or server cluster that is different from the processing device 3 and can communicate with the transformer internal inspection device 2 and / or the processing device 3. Correspondingly, the transformer internal health status monitoring device based on the internal inspection robot provided in the embodiment of the present invention can also be arranged in a server or server cluster that is different from the processing device 3 and can communicate with the transformer internal inspection device 2 and / or the processing device 3.

[0031] For example, the initial image can be collected by the transformer internal inspection device 2, and then the transformer internal inspection device 2 can send the initial image to the processing device 3, and the processing device 3 that receives the initial image executes the transformer internal health status monitoring method based on the internal inspection robot provided in an embodiment of the present invention.

[0032] It should be understood that Figure 1 The number of transformer internal inspection devices, networks and processing devices in the figure is only for illustration. Any number of internal inspection devices, networks and processing devices may be provided according to implementation requirements.

[0033] Figure 2 A flow chart of a defect target detection method for a robot fish operation inside a transformer according to an embodiment of the present invention is shown.

[0034] like Figure 2 As shown, the method includes operations S201 to S205.

[0035] In operation S201 , in response to a transformer health status monitoring request, a plurality of initial images captured by an internal inspection robot of the transformer are acquired.

[0036] According to an embodiment of the present invention, the health status of the transformer may include status information such as the electrical performance, insulation performance and structural integrity of the transformer, wherein the electrical performance of the transformer may, for example, include the stability of electrical parameters such as voltage, current, power factor, etc. of the transformer, the insulation performance of the transformer may, for example, include partial discharge results, and the structural integrity of the transformer may include the integrity and stability of components such as windings, cores, and oil tanks.

[0037] According to an embodiment of the present invention, a plurality of initial images may be acquired based on an image acquisition sensor configured on the internal inspection robot.

[0038] In operation S202, feature matching is performed based on the multiple initial images to obtain key point matching results between the multiple initial images.

[0039] According to an embodiment of the present invention, key points of each of the multiple initial images are identified and extracted, and the extracted multiple key points are matched by an image feature matching method to obtain key point matching results between the multiple initial images.

[0040] According to an embodiment of the present invention, key point matching results between multiple initial images may be used to determine relative positions and relationships between the multiple initial images.

[0041] In operation S203, depth alignment and splicing are performed on the multiple initial images based on the key point matching results between the multiple initial images to obtain a first spliced ​​image.

[0042] According to an embodiment of the present invention, the multiple initial images are spliced ​​based on the relative positions and relationships between the multiple initial images, so as to obtain a more complete and coherent image, namely, a first spliced ​​image.

[0043] According to an embodiment of the present invention, before stitching the multiple initial images, depth alignment may be performed on the multiple initial images to ensure that image regions at different depths can be accurately aligned based on the depth differences between the multiple initial images.

[0044] In operation S204, rectangularization correction is performed on the first stitched image to obtain a second stitched image.

[0045] According to the embodiment of the present invention, since the first stitched image may have irregular boundaries, the first stitched image may be subjected to boundary correction processing based on a rectangularization correction method to obtain a second stitched image with regular boundaries.

[0046] In operation S205 , a health status detection is performed based on the second stitched image to obtain an internal health status monitoring result of the transformer.

[0047] According to an embodiment of the present invention, an image processing and analysis method can be used to perform health status detection on the second spliced ​​image to identify potential problems inside the transformer, such as insulation damage, oil leakage, overheating, etc., and generate corresponding internal health status monitoring results. The image processing and analysis method can include operations such as image segmentation, feature extraction, classification, and recognition.

[0048] Based on this, the embodiment of the present invention uses feature matching and depth alignment methods to accurately capture key feature points in the image, and achieves precise alignment between images based on these key feature points. This method not only takes into account the overall transformation of the image, but also takes into account the adjustment of local details, thereby reducing ghosting and blurring problems. In addition, by further correcting the image boundaries and details to naturally process the transition of the image boundaries, the spliced ​​image is smoother and more natural at the boundaries, thereby providing strong data support for transformer health status monitoring.

[0049] Figure 3 A flow chart of a method for determining key point matching results between multiple initial images according to an embodiment of the present invention is shown.

[0050] like Figure 3 As shown, the method includes operations S301 to S304.

[0051] In operation S301 , for each initial image, a key point heat map, a dense feature description map, and a reliability heat map are extracted from the initial image.

[0052] In operation S302, a plurality of candidate key points are determined from a key point heat map of the initial image based on the reliability heat map of the initial image.

[0053] In operation S303, first local features corresponding to each of the plurality of candidate feature points are determined from a dense feature description map of the initial image.

[0054] In operation S304, feature matching is performed based on the plurality of first local features of each of the plurality of initial images to obtain key point matching results between the plurality of initial images.

[0055] According to an embodiment of the present invention, a key point heat map can be used to display multiple key point positions in each initial image. The key point heat map can be a two-dimensional matrix, each element in the matrix corresponds to a pixel in the image, and the value of each element represents the probability or intensity of the pixel position being a key point. In the key point heat map, the position and intensity of the key point can be represented by color coding (such as red for high intensity and blue for low intensity) or grayscale values.

[0056] According to a specific embodiment of the present invention, a lightweight image matching (XFeat) algorithm can be used to obtain a key point heat map. The algorithm achieves extremely high speed through early downsampling and shallow convolution, and introduces deeper convolution processing in the subsequent encoder to obtain multiple feature points, further enhancing the accuracy and robustness of feature extraction. Among them, a parallel branch can be introduced in the XFeat algorithm to capture the key points in the initial image.

[0057] For example, the initial image can be gridded and represented as The 2D grid of pixels is used to divide the initial image into smaller regions to capture the location information of key points more finely. Each grid cell is reshaped to obtain 64-dimensional features, ensuring that each grid cell contains enough information to maintain spatial resolution. Convolution is used to regress the coordinates of key points. After processing through four convolutional layers, a key point heat map can be obtained, where the key point heat map can include the logarithmic scores of the key point distribution, which can be used to reflect the possibility of the existence of key points at various positions in the initial image. Each key point is classified into one of the 64 possible positions within the grid cell, plus an invalid point class to handle the situation where the key point is not found, ensuring that the key points in the image are effectively detected and matched with limited computing resources. According to the positions of multiple key points in the key point heat map and the corresponding scores, the specific positions of the key points can be determined.

[0058] According to an embodiment of the present invention, a dense feature description map can be used to display the features of each pixel or region in an image. The dense feature description map can also be a two-dimensional matrix, and each element in the matrix can represent a feature vector of the pixel or region. The feature vector can include multiple dimensions to describe multiple feature attributes of the pixel or region, such as color, texture, shape, etc.

[0059] According to a specific embodiment of the present invention, a dense feature description graph may be obtained based on operations such as multi-scale feature extraction, feature upsampling and projection, and convolution fusion.

[0060] For example, the descriptor head can be used to extract multi-scale features from the encoder, and the receptive field of view can be gradually expanded through continuous convolution blocks to capture a wider range of image information. In this process, the resolution of the feature map will gradually decrease until it reaches 1 / 32 of the original resolution. This design helps to enhance the robustness of the model to viewpoint changes. Even if the position or perspective of objects in the image changes, the model can still extract stable feature information. After obtaining the multi-scale features, feature upsampling and projection are performed. Upsampling can improve the resolution of feature maps by interpolation or deconvolution. Among them, the intermediate feature representation can be upsampled proportionally, and the specific ratios include 1 / 8, 1 / 16 and 1 / 32. The upsampled feature map will be projected to a specific spatial size, that is, , where 𝐻 and 𝑊 are the height and width of the input image, respectively. This projection operation helps to unify feature maps of different scales to the same spatial resolution. Finally, through the convolution fusion operation. The upsampled and projected multi-scale representations are merged into a dense feature description map. Among them, the convolution fusion block can include multiple convolution layers and pooling layers to further process and fuse the input multi-scale features. Through the convolution operation, the network can learn the correlation between features of different scales and integrate them into a unified dense feature description map. The dense feature description map not only contains rich image information, but also has strong robustness and discriminability, and can be used for subsequent feature matching and recognition tasks.

[0061] According to an embodiment of the present invention, a reliability heat map can be used to display the confidence or reliability of feature matching. The reliability heat map can also be a two-dimensional matrix, each element in the matrix corresponds to a pixel or feature point in the image, and the value of each element can represent the confidence or reliability of the matching result of the pixel or feature point. The area with high confidence can indicate that the matching result is more accurate, while the area with low confidence can indicate that there is an error in the matching result. The confidence or reliability of the pixel or feature point matching result can also be represented by color coding or grayscale value.

[0062] For example, the obtained multi-scale features can be sent to one or more additional convolution blocks, and the reliability heat map can be obtained after being processed by the additional convolution blocks. These additional convolution blocks can include convolution layers, activation functions (such as ReLU), batch normalization layers, etc., for further processing and refining features.

[0063] According to an embodiment of the present invention, a plurality of candidate key points are identified and extracted from a key point heat map of an initial image. Specifically, the top K image regions with the highest reliability scores can be identified from the reliability heat map and temporarily stored for use in subsequent steps, thereby controlling memory and computational costs and avoiding processing the entire image. The key point heat map is traversed, and for each key point, the key point with a higher score is selected as a candidate key point by calculating the score of each key point, and the position coordinates of each candidate key point are recorded. Among them, the key point score can be calculated based on the confidence score of the key point and the reliability score of the key point, and the key point score is calculated as follows:

[0064] (1);

[0065] In the formula, score represents the key point score, represents the confidence score of the key point, Represents the reliability score of the key point.

[0066] According to an embodiment of the present invention, a first local feature corresponding to each of the plurality of candidate key points is determined from a dense feature description map of an initial image. Specifically, for each candidate key point, a corresponding local feature region is found in the dense feature description map according to its position coordinates, and a first local feature is extracted from the local feature region. The first local feature may be a feature vector of a fixed length.

[0067] According to an embodiment of the present invention, feature matching is performed based on multiple first local features of each of the multiple initial images to obtain key point matching results between the multiple initial images, including: for each initial image, interpolating the multiple first local features of the initial image to obtain multiple second local features; using a mutual nearest neighbor search method to match the multiple second local features of the multiple initial images to obtain an initial matching result; based on the initial matching result, densely matching the multiple second local features of the multiple initial images to obtain key point matching results between the multiple initial images.

[0068] According to an embodiment of the present invention, in order to improve the accuracy of feature matching, interpolation processing may be performed on the first local features corresponding to each of the plurality of candidate key points to obtain a second local feature with a higher resolution.

[0069] For example, for each initial image, bicubic interpolation can be used to interpolate multiple first local features to obtain multiple second local features. Bicubic interpolation can generate a smoother image than bilinear interpolation while retaining more detail information. The obtained second local features also have higher resolution and accuracy.

[0070] According to an embodiment of the present invention, a mutual nearest neighbor (MNN) search method may be used to match a plurality of second local features of a plurality of initial images to obtain an initial matching result, wherein the initial matching result may include one or more matching pairs.

[0071] For example, for each second local feature, find the closest feature in the feature set of another initial image, and determine whether this closest feature also takes the original feature as its closest feature. If so, they are considered to be a pair of matching features.

[0072] According to the embodiment of the present invention, since the initial matching result may include some erroneous matches or missed matches, further matching may be performed on the plurality of second local features of the plurality of initial images based on the initial matching result.

[0073] Specifically, in a resource-constrained environment, a strategy from coarse matching to fine matching can be used to achieve semi-dense matching, and in the process, the use of high-resolution feature maps can be avoided. Specifically, a lightweight multi-layer perceptron (MLP) can be used to predict pixel-level offsets. The MLP is used to fine-tune the initial matching results to obtain key point matching results between multiple initial images to predict pixel-level offsets, thereby further improving the accuracy of matching.

[0074] Based on this, the embodiment of the present invention uses a feature point extraction and matching method, based on the extracted key point heat map, dense feature description map and reliability heat map, to accurately extract a large number of stable and distinguishing key points in complex scenes, ensuring the robustness and stability of the feature points under different viewing angles, scales and lighting conditions. In addition, in the feature matching stage, by interpolating multiple first local features of multiple initial images, performing mutual nearest neighbor search and dense matching, etc., not only the resolution and accuracy of the features are improved, but also the key point positions can be accurately adjusted to achieve pixel-level matching accuracy.

[0075] According to an embodiment of the present invention, multiple initial images are deeply aligned and stitched based on key point matching results between the multiple initial images to obtain a first stitched image, including: determining a reference image from the multiple initial images; for each other initial image in the multiple other initial images, determining the association relationship between the other initial images and the reference image, the other initial images include initial images other than the reference image in the multiple initial images; based on the association relationship between the other initial images and the reference image and the key point matching results between the multiple initial images, stitching the other initial images to the reference image space where the reference image is located; and traversing and processing the multiple other initial images to obtain the first stitched image.

[0076] According to an embodiment of the present invention, one of the initial images is selected from the multiple initial images as a reference image to serve as a starting point and reference for stitching images, wherein the reference image may be a background image or a foreground image.

[0077] According to an embodiment of the present invention, for each other initial image, the association relationship between the other initial image and the benchmark image can be used to characterize the overlapping area between each other initial image and the benchmark image, wherein the overlapping area can be determined based on the key point matching results between the other initial image and the benchmark image.

[0078] According to an embodiment of the present invention, based on the association relationship between the other initial images and the reference image and the key point matching result, the other initial images are spliced ​​into the reference image space where the reference image is located. In the splicing process, the other initial images can be spliced ​​using image transformation and image fusion technology. For example, pixel mapping can be performed on the overlapping area, and isomorphic transformation can be performed on the non-overlapping area.

[0079] According to an embodiment of the present invention, the above steps are repeated for all other initial images until all other initial images are spliced ​​into the reference image space to form a first spliced ​​image.

[0080] According to an embodiment of the present invention, based on the association relationship between other initial images and the benchmark image and the key point matching results between multiple initial images, other initial images are spliced ​​into the benchmark image space where the benchmark image is located, including: when the association relationship between other initial images and the benchmark image is represented by the existence of an overlapping area between the other initial images and the benchmark image, based on multiple first key points included in the other initial images and multiple second key points included in the benchmark image, a first affine transformation matrix and a second affine transformation matrix are determined; based on the first affine transformation matrix and the second affine transformation matrix, pixel points located in the overlapping area in the other initial images are mapped to the benchmark image space to obtain a first image area; pixel points in the other initial images that are not located in the overlapping area are subjected to isomorphic transformation to obtain a second image area located in the benchmark image space; the first image area, the second image area and the benchmark image are spliced ​​to splice the other initial images into the benchmark image space where the benchmark image is located.

[0081] According to an embodiment of the present invention, when there is an overlapping area between the other initial image and the reference image, an affine transformation matrix can be used to map pixels in one image to the coordinate system of another image to ensure that the pixels in the overlapping part can be seamlessly merged. However, in some complex cases, a single affine transformation may not accurately describe the transformation relationship between the two images. Therefore, a double affine transformation method can be used to distort and construct a spliced ​​image.

[0082] According to an embodiment of the present invention, the multiple first key points included in the other initial images can be used to characterize the pixels with significant features in the other initial images. The multiple second key points included in the reference image can be used to characterize the pixels with significant features for reference.

[0083] According to an embodiment of the present invention, key points for generating a first affine transformation matrix and a second affine transformation matrix are respectively determined based on distances between a plurality of first key points included in other initial images and a plurality of second key points included in the reference image.

[0084] According to an embodiment of the present invention, the pixels in the overlapping area of ​​the other initial images are mapped to the reference image space using the first affine transformation matrix and the second affine transformation matrix, so that the pixels in the overlapping area can be aligned with the corresponding pixels in the reference image. The mapped pixel set can constitute the first image area.

[0085] According to an embodiment of the present invention, pixel points in other initial images that are not located in the overlapping area may be subjected to isomorphic transformation to ensure that they can smoothly transition to the reference image space, and the set of pixel points after the isomorphic transformation may constitute the second image area.

[0086] According to an embodiment of the present invention, the first image region (e.g., the mapping result of the overlapping region) and the second image region (e.g., the transformation result of the non-overlapping region) are spliced ​​with the reference image to obtain an image splicing result. During the splicing process, the spliced ​​image may be subjected to steps such as color correction and brightness adjustment to ensure that the spliced ​​image is natural and coherent.

[0087] According to an embodiment of the present invention, based on multiple first key points included in other initial images and multiple second key points included in the reference image, a first affine transformation matrix and a second affine transformation matrix are determined, including: determining a first target key point and a second target key point from multiple first key points and multiple second key points based on the average value of the distance between each key point in the multiple first key points and the multiple second key points and multiple other key points; clustering the multiple first key points and the multiple second key points with the first target key point and the second target key point as clustering centers to obtain a first initial clustering cluster and a second initial clustering cluster; performing random sample consensus calculation based on the first initial clustering cluster and the second initial clustering cluster, respectively, to obtain a first affine transformation matrix and a second affine transformation matrix.

[0088] According to an embodiment of the present invention, based on multiple first key points included in other initial images and multiple second key points included in the reference image, the average distance between each key point and all other key points is calculated, and the key points with the shortest average distance and the longest average distance are selected as target key points. For example, the key point with the shortest average distance can be used as the first target key point, and the key point with the longest average distance can be used as the second target key point.

[0089] According to an embodiment of the present invention, a first target key point and a second target key point are respectively used as two cluster centers to perform clustering processing on a plurality of first key points and second key points.

[0090] For example, a k-means clustering algorithm may be used to cluster the first key points and the second key points to obtain two groups of initial clusters. The first initial cluster may include key points that are closer to the first target key points, mainly from other initial images. The second initial cluster may include key points that are closer to the second target key points, mainly from the reference image.

[0091] According to an embodiment of the present invention, for each group of initial clusters, a first affine transformation matrix and a second affine transformation matrix can be calculated using a random sample consensus (RANSAC) algorithm. The random sample consensus algorithm can be used to estimate the parameters of the model from data containing noise and outliers, and is suitable for processing false matches in feature point matching.

[0092] According to an embodiment of the present invention, based on a first affine transformation matrix and a second affine transformation matrix, pixel points in other initial images located in the overlapping area are mapped to a reference image space to obtain a first image area, including: removing outliers from the first initial cluster cluster based on the first affine transformation matrix to obtain a first cluster cluster; removing outliers from the second initial cluster cluster based on the second affine transformation matrix to obtain a second cluster cluster; for each pixel point in the overlapping area in other initial images, determining a mapping weight of the pixel point based on a first distance from the pixel point to the first cluster cluster and a second distance from the pixel point to the second cluster cluster; and mapping the pixel point to the reference image space based on the mapping weight of the pixel point, the first affine transformation matrix and the second affine transformation matrix to obtain the first image area.

[0093] According to an embodiment of the present invention, outliers are removed from the first initial cluster cluster based on the first affine transformation matrix to obtain a first cluster cluster. The removed outliers may include key feature points that are inconsistent with the affine transformation matrix due to erroneous feature point matching or noise. Similarly, outliers are removed from the second initial cluster cluster based on the second affine transformation matrix to obtain a second cluster cluster.

[0094] According to an embodiment of the present invention, for each pixel point in the other initial images located in the overlapping area, the mapping weight of each pixel point can be determined by the distance from the pixel point to the first cluster and the second cluster respectively.

[0095] Specifically, the first distance can be used to characterize the Euclidean distance from each pixel to the nearest feature point in the first cluster, and the second distance can be used to characterize the Euclidean distance from each pixel to the nearest feature point in the second cluster. Based on the calculated first distance and second distance, the mapping weight of each pixel can be calculated as follows:

[0096] (2);

[0097] In the formula, Represents the mapping weight of each pixel, Represents the inverse of the Euclidean distance from each pixel to the nearest feature point in the first cluster. Represents the inverse of the Euclidean distance from each pixel to the nearest feature point in the second cluster.

[0098] According to an embodiment of the present invention, for each pixel point, its position mapped to the reference image space can be calculated by weighted fusion of the results of two affine transformation matrices. The specific calculation method is as follows:

[0099] (3);

[0100] In the formula, Represents the affine transformation matrix of each pixel, Represents the mapping weight of each pixel, which is used to control the weight of the affine transformation matrix of each pixel at the pixel position (i, j). represents the first affine transformation matrix, Represents the second affine transformation matrix.

[0101] According to an embodiment of the present invention, based on the affine transformation matrix of each pixel point, the corresponding positions of the pixel points in the overlapping area of ​​other initial images in the reference image space can be obtained, and each pixel point is mapped to the reference image space to construct the first image area.

[0102] Based on this, the embodiment of the present invention is based on the image alignment method of the dual affine transformation matrix, which aligns different depth areas in the image, such as the foreground plane and the background plane, respectively, and can effectively deal with the depth difference problem in the image, thereby improving the accuracy of image stitching. In addition, the use of clustering processing and random sample consensus algorithm can remove abnormal points, thereby enhancing the robustness of image stitching and reducing stitching errors caused by incorrect matching or noise points.

[0103] According to an embodiment of the present invention, based on the association relationship between other initial images and the benchmark image and the key point matching results between the multiple initial images, the other initial images are spliced ​​into the image space where the initial images are located, including: when the association relationship between the other initial images and the benchmark image is represented as the other initial images and the benchmark image not overlapping, and there are n intermediate images between the other initial images and the benchmark image, the overlapping area between the other initial images and the first intermediate image is sequentially mapped to the image space where the n intermediate images are located, until the overlapping area between the other initial images and the first intermediate image is mapped to the benchmark image. A third image area in a quasi-image space, wherein there is an overlapping area between the nth intermediate image and the reference image; performing isomorphic transformation on pixel points in non-overlapping areas between other initial images and the first intermediate image, so as to sequentially map the non-overlapping areas between the other initial images and the first intermediate image to the image spaces where the n intermediate images are located, until the non-overlapping areas between the other initial images and the first intermediate image are mapped to the fourth image area in the reference image space; and splicing the third image area, the fourth image area and the reference image, so as to splice the other initial images to the reference image space where the reference image is located.

[0104] According to an embodiment of the present invention, when the association relationship between other initial images and the benchmark image is expressed as the other initial images not overlapping with the benchmark image, and there are n intermediate images between the other initial images and the benchmark image, for each other initial image, the transformation matrix that maps the overlapping area from the other initial image to the first intermediate image space can be calculated based on the double affine transformation matrix, and this process can be repeated to map the overlapping area to subsequent intermediate image spaces in sequence until the benchmark image space is reached to form a third image area.

[0105] Specifically, an image is selected as a reference image, and all other initial images will be mapped onto this reference image plane. For each pair of adjacent overlapping images, an affine transformation matrix between them is calculated based on a double affine transformation matrix to map the overlapping area from one image to a virtual representation of another image. When processing an image adjacent to a reference image, the affine transformation matrix from the image to the reference image can be directly used. When processing an image overlapping with a reference image, the overlapping area can be directly mapped onto the reference image.

[0106] For example, for three overlapping images I0, I1, and I2, where I0 can be used as the reference image, the double affine transformation matrix that maps image I1 to image I0 can be calculated and expressed as Then calculate the double affine transformation matrix that maps image I2 to image I1 and express it as Since image I1 has been mapped to the reference image, image I2 needs to be mapped to the reference image as well. In the overlapping area of ​​image I2 and image I1, you can use Map image I2 into the coordinate system of image I1, and then use This mapped image is further mapped onto the reference image to associate image I2 with the reference image.

[0107] According to an embodiment of the present invention, pixel points in non-overlapping areas between other initial images and the first intermediate image are subjected to isomorphic transformation so as to sequentially map the non-overlapping areas between other initial images and the first intermediate image to the image spaces where the n intermediate images are located respectively, until the non-overlapping areas between the other initial images and the first intermediate image are mapped to the fourth image area in the reference image space.

[0108] In the above specific embodiment, for the non-overlapping area of ​​image I2 and image I1, since it is not possible to directly use , so we can use the isomorphic transformation of the overlapping boundary of image I2 and image I1 to expand The overlapping boundary for filling can be obtained by weighting the isomorphic transformation of all points on the boundary until the non-overlapping area between the other initial images and the first intermediate image is mapped to the fourth image area in the reference image space. The closer the distance from the pixel on the image I2 to the boundary point is, the greater the weight is. Specifically, the calculation formula is as follows:

[0109] (4);

[0110] In the formula, Represents the double affine transformation matrix mapping image I2 to the reference image, Represents the double affine transformation matrix that maps image I1 to the reference image The double affine transformation matrix expression that maps image I2 to the reference image is, where, represents the overlapping boundary, p represents the pixel point on the image I2, q represents the boundary point on the overlapping boundary, Represents a weight scalar, which can be obtained by calculating the Euclidean distance. The weight scalar can be normalized as .

[0111] According to an embodiment of the present invention, the third image area, such as the overlapping area, and the fourth image area, such as the non-overlapping area, are spliced ​​with the reference image respectively to splice other initial images to the reference image space where the reference image is located, ensuring that the transition at the splicing point is smooth and natural.

[0112] According to an embodiment of the present invention, a first stitched image is rectangularized to obtain a second stitched image, including: adding noise to initial motion field data to obtain a noisy motion field; denoising the noisy motion field using the first stitched image and an image mask as constraints to obtain a restored motion field; deforming the first stitched image based on the restored motion field to obtain a rectangularized third stitched image; and weighted sampling the third stitched image based on a confidence mask, a consistency constraint, and a authenticity constraint to obtain a second stitched image.

[0113] According to an embodiment of the present invention, due to different shooting angles, the image boundary generated by the homography-based alignment method is no longer a rectangle. Therefore, a motion diffusion model (MDM) can be used to convert the stitched image with irregular edges and white edges into a seamless rectangular format without these edges.

[0114] Specifically, in order to increase the robustness of the model and prevent overfitting, the initial motion field data can be doped with noise. For example, the initial motion field data is sampled, and then a time step is randomly selected and noise is added to obtain a noisy motion field. The intensity and type of noise (such as Gaussian noise, Poisson noise, etc.) can be set according to actual conditions. Gaussian noise can be injected into the original data distribution in sequence. , to generate an image with gradually increasing noise, where the expression of this process is as follows:

[0115] (5);

[0116] (6);

[0117] (7);

[0118] In the formula, represents the Gaussian distribution of the motion field, x0 represents the initial motion field data, x t represents the noisy motion field data, represents the Gaussian distribution function, represents the mean value of the weight coefficient, I represents the noise image, represents the i-th weight coefficient, t represents the time step, represents the weight coefficient, represents a hyperparameter, and the hyperparameter increases gradually with each time step.

[0119] In this specific embodiment, the first spliced ​​image and the image mask can be used as constraints to perform denoising on the noisy motion field, and the noisy motion field, the first spliced ​​image and the image mask are input into the denoising model together, and the noise is removed by the learning ability of the model to obtain the restored motion field. Specifically, in order to strengthen the model's control over the generation process and improve the fidelity of the generated image, additional adjustment variables can be introduced, and the adjustment mechanism combines these variables with the intermediate noise data to obtain better results. Among them, the expression of the noise distribution model for each time interval is as follows:

[0120] (8);

[0121] In the formula, represents the denoising model, x t-1 represents the restored motion field, y represents the adjustment variable, Express expectations, Represents variance.

[0122] In this specific embodiment, a deformation function can be used to deform the first stitched image based on the restored motion field to obtain a rectangular third stitched image. The calculation method is as follows:

[0123] (9);

[0124] In the formula, represents the rectangularized third stitched image, represents the distortion function, represents the first stitched image, Represents the generated motion field.

[0125] According to an embodiment of the present invention, a content diffusion model (CDM) may be used to refine the generated rectangularization result, thereby generating an optimized rectangularization image.

[0126] Specifically, the rectangular image can be sampled and refined using an “image-to-image” diffusion process. Similar to MDM, the distribution of the target optimization data is first sampled, then noise is added and input into the noise model, which outputs the optimized rectangular image.

[0127] In this specific embodiment, in order to deal with the noise and artifacts that may be introduced, a confidence mask can be introduced to guide the sampling process. For example, the confidence mask can be calculated based on the white edge mask, the intensity map of the motion field, and the input spliced ​​image mask. The higher the value of the confidence mask, the better the image quality of the corresponding area, and the more reliable it is. The confidence mask is calculated as follows:

[0128] (10);

[0129] Where M R represents the confidence mask, M1 represents the white edge mask, M0 represents the intensity of the motion field, and M S represents the third stitched image mask of the input, represents a hyperparameter.

[0130] In this specific embodiment, the consistency constraint and the authenticity constraint can also be combined to use weighted sampling technology to gradually reduce noise and enhance the authenticity of the image. The expressions of the consistency constraint and the authenticity constraint are as follows:

[0131] (11);

[0132] (12);

[0133] In the formula, represents the initial rectangular image, represents an optimized rectangular image, represents the distribution of target optimization data, and A represents the reference matrix.

[0134] In this specific embodiment, the consistency constraint and the authenticity constraint can be combined into the sampling process based on the rank-zero degree theorem (RNT). The specific expression is as follows:

[0135] (13);

[0136] In the formula, represents the pseudo-inverse of the reference matrix A. This formula can express the truth constraint as a combination of its projection in the range space of the reference matrix A and its projection in the corresponding null space.

[0137] By calculating the weighted sum of the confidence mask, the initial rectangular image, and the output of each iteration of CDM, the noise is gradually reduced and the authenticity of the image is enhanced, thereby generating a geometrically accurate and visually pleasing rectangular image. Specifically, the expression of the optimized rectangular image is as follows:

[0138] (14);

[0139] In the formula, represents the output of each iteration of CDM, Represents the mean value of weight coefficient.

[0140] Figure 4 A schematic diagram of the internal image of a transformer after processing according to a specific embodiment of the present invention is shown.

[0141] like Figure 4As shown, the initial image 401 is acquired by the transformer internal inspection robot, and the key points in the initial image 401 are detected and matched using the lightweight image matching (XFeat) algorithm to obtain the key point detection and matching image 402. Based on the key point matching results between the initial images, the initial images are deeply aligned and spliced ​​to obtain the spliced ​​image 403. The spliced ​​image 403 is rectangularized and corrected to obtain the corrected image 404.

[0142] Figure 5 A block diagram of a transformer internal health status monitoring device based on an internal inspection robot according to an embodiment of the present invention is shown.

[0143] like Figure 5 As shown, the transformer internal health status monitoring device includes an acquisition module 510 , a matching module 520 , a splicing module 530 , a correction module 540 and a detection module 550 .

[0144] The acquisition module 510 is used to acquire a plurality of initial images captured by an internal inspection robot of the transformer in response to a transformer health status monitoring request.

[0145] The matching module 520 is used to perform feature matching based on the multiple initial images to obtain key point matching results between the multiple initial images.

[0146] The stitching module 530 is used to perform depth alignment and stitching on the multiple initial images based on the key point matching results between the multiple initial images to obtain a first stitched image.

[0147] The correction module 540 is used to perform rectangular correction on the first stitched image to obtain a second stitched image.

[0148] The detection module 550 is used to perform health status detection based on the second spliced ​​image to obtain the internal health status monitoring result of the transformer.

[0149] According to an embodiment of the present invention, the matching module 520 includes an extraction submodule, a first determination submodule, a second determination submodule, and a matching submodule.

[0150] The extraction submodule is used to extract a key point heat map, a dense feature description map and a reliability heat map from each initial image.

[0151] The first determination submodule is used to determine a plurality of candidate key points from the key point heat map of the initial image based on the reliability heat map of the initial image.

[0152] The second determination submodule is used to determine the first local features corresponding to each of the plurality of candidate key feature points from the dense feature description graph of the initial image.

[0153] The matching submodule is used to perform feature matching based on the multiple first local features of each of the multiple initial images to obtain key point matching results between the multiple initial images.

[0154] According to an embodiment of the present invention, the matching submodule includes an interpolation unit, a matching unit, and a dense matching unit.

[0155] The interpolation unit is used to perform interpolation processing on multiple first local features of each initial image to obtain multiple second local features.

[0156] The matching unit is used to match the plurality of second local features of the plurality of initial images by using a mutual nearest neighbor search method to obtain an initial matching result.

[0157] The dense matching unit is used to perform dense matching on the multiple second local features of the multiple initial images based on the initial matching result to obtain key point matching results between the multiple initial images.

[0158] According to an embodiment of the present invention, the splicing module 530 includes a third determining submodule, a fourth determining submodule, a splicing submodule, and a traversing submodule.

[0159] The third determination submodule is used to determine a reference image from a plurality of initial images.

[0160] The fourth determination submodule is used to determine, for each of the multiple other initial images, an association relationship between the other initial images and the reference image, where the other initial images include the initial images other than the reference image in the multiple initial images.

[0161] The splicing submodule is used to splice other initial images into the reference image space where the reference image is located based on the association relationship between the other initial images and the reference image and the key point matching results between the multiple initial images.

[0162] The traversal submodule is used to traverse and process multiple other initial images to obtain a first stitched image.

[0163] According to an embodiment of the present invention, the splicing submodule includes a first determining unit, a first mapping unit, an isomorphic transformation unit, and a first splicing unit.

[0164] The first determination unit is used to determine the first affine transformation matrix and the second affine transformation matrix based on multiple first key points included in the other initial images and multiple second key points included in the benchmark image, when the association relationship between the other initial images and the benchmark image is represented by the existence of an overlapping area between the other initial images and the benchmark image.

[0165] The first mapping unit is used to map the pixel points in the overlapping area of ​​other initial images to the reference image space based on the first affine transformation matrix and the second affine transformation matrix to obtain the first image area.

[0166] The isomorphic transformation unit is used to perform isomorphic transformation on the pixel points in other initial images that are not located in the overlapping area, so as to obtain a second image area located in the reference image space.

[0167] The first splicing unit is used to splice the first image area, the second image area and the reference image, so as to splice other initial images into the reference image space where the reference image is located.

[0168] According to an embodiment of the present invention, the first determining unit includes a first determining subunit, a clustering subunit, and a calculating subunit.

[0169] The first determination subunit is used to determine a first target key point and a second target key point from the plurality of first key points and the plurality of second key points based on an average value of distances between each key point in the plurality of first key points and the plurality of second key points and a plurality of other key points.

[0170] The clustering subunit is used to cluster the first target key point and the second target key point as cluster centers, and to obtain a first initial clustering cluster and a second initial clustering cluster.

[0171] The calculation subunit is used to perform random sample consensus calculation based on the first initial clustering cluster and the second initial clustering cluster respectively to obtain a first affine transformation matrix and a second affine transformation matrix.

[0172] According to an embodiment of the present invention, the first mapping unit includes a first eliminating subunit, a second eliminating subunit, a second determining subunit, and a mapping subunit.

[0173] The first elimination subunit is used to eliminate outliers from the first initial cluster based on the first affine transformation matrix to obtain a first cluster.

[0174] The second elimination subunit is used to eliminate outliers from the second initial clustering cluster based on the second affine transformation matrix to obtain a second clustering cluster.

[0175] The second determining subunit is used to determine, for each pixel point in the overlapping area in other initial images, a mapping weight of the pixel point based on a first distance from the pixel point to the first cluster and a second distance from the pixel point to the second cluster cluster.

[0176] The mapping subunit is used to map the pixel points to the reference image space based on the mapping weights of the pixel points, the first affine transformation matrix and the second affine transformation matrix to obtain the first image area.

[0177] According to an embodiment of the present invention, the splicing submodule includes a second mapping unit, a transform unit, and a second splicing unit.

[0178] The second mapping unit is used to map the overlapping area between the other initial images and the first intermediate image in sequence to the image space where the n intermediate images are located, when the association relationship between the other initial images and the benchmark image is represented by the other initial images not overlapping with the benchmark image and there are n intermediate images between the other initial images and the benchmark image, until the overlapping area between the other initial images and the first intermediate image is mapped to the third image area in the benchmark image space.

[0179] A transformation unit is used to perform isomorphic transformation on pixel points in non-overlapping areas between other initial images and the first intermediate image, so as to sequentially map the non-overlapping areas between other initial images and the first intermediate image to the image spaces where the n intermediate images are located, until the non-overlapping areas between the other initial images and the first intermediate image are mapped to the fourth image area in the reference image space.

[0180] The second stitching unit is used to stitch the third image area, the fourth image area and the reference image, so as to stitch other initial images into the reference image space where the reference image is located.

[0181] According to an embodiment of the present invention, the correction module 540 includes a doping submodule, a denoising submodule, a deformation submodule, and a sampling submodule.

[0182] The doping submodule is used to perform noise doping on the initial motion field data to obtain a noisy motion field.

[0183] The denoising submodule is used to perform denoising processing on the noisy motion field by taking the first spliced ​​image and the image mask as constraints to obtain a restored motion field.

[0184] The deformation submodule is used to perform deformation processing on the first spliced ​​image based on the restored motion field to obtain a rectangular third spliced ​​image.

[0185] The sampling submodule is used to perform weighted sampling on the third stitched image based on the confidence mask, the consistency constraint and the authenticity constraint to obtain the second stitched image.

[0186] According to the embodiments of the present invention, any one or more of the modules, submodules, units, and subunits, or at least part of the functions of any one of them can be implemented in one module. According to the embodiments of the present invention, any one or more of the modules, submodules, units, and subunits can be split into multiple modules for implementation. According to the embodiments of the present invention, any one or more of the modules, submodules, units, and subunits can be at least partially implemented as hardware circuits, such as field programmable gate arrays (FPGAs), programmable logic arrays (PLAs), systems on chips, systems on substrates, systems on packages, application specific integrated circuits (ASICs), or can be implemented by hardware or firmware in any other reasonable way of integrating or packaging the circuit, or by any one of the three implementation methods of software, hardware, and firmware, or by a proper combination of any of them. Alternatively, according to the embodiments of the present invention, one or more of the modules, submodules, units, and subunits can be at least partially implemented as computer program modules, and when the computer program modules are run, the corresponding functions can be executed.

[0187] For example, any multiple of the acquisition module 510, the matching module 520, the splicing module 530, the correction module 540, and the detection module 550 can be combined in one module / unit / sub-unit for implementation, or any one of the modules / units / sub-units can be split into multiple modules / units / sub-units. Alternatively, at least part of the functions of one or more of these modules / units / sub-units can be combined with at least part of the functions of other modules / units / sub-units and implemented in one module / unit / sub-unit. According to an embodiment of the present invention, at least one of the acquisition module 510, the matching module 520, the splicing module 530, the correction module 540, and the detection module 550 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or by any one of the three implementation methods of software, hardware, and firmware, or by a suitable combination of any of them. Alternatively, at least one of the acquisition module 510, the matching module 520, the splicing module 530, the correction module 540 and the detection module 550 may be at least partially implemented as a computer program module, and when the computer program module is executed, the corresponding function may be executed.

[0188] It should be noted that the transformer internal health status monitoring device part in the embodiment of the present invention corresponds to the transformer internal health status monitoring method part in the embodiment of the present invention. The description of the transformer internal health status monitoring device part specifically refers to the transformer internal health status monitoring method part, which will not be repeated here.

[0189] Figure 6 A block diagram of an electronic device suitable for implementing a transformer internal health status monitoring method based on an internal inspection robot according to an embodiment of the present invention is shown. Figure 6 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0190] like Figure 6 As shown, the electronic device according to an embodiment of the present invention includes a processor 601, which can perform various appropriate actions and processes according to the program stored in the read-only memory ROM 602 or the program loaded from the storage part 608 to the random access memory RAM 603. The processor 601 may include, for example, a general-purpose microprocessor (such as a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (for example, an application-specific integrated circuit (ASIC)), etc. The processor 601 may also include an on-board memory for caching purposes. The processor 601 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present invention.

[0191] In RAM 603, various programs and data required for the operation of the electronic device are stored. The processor 601, ROM 602 and RAM 603 are connected to each other via a bus 604. The processor 601 performs various operations of the method flow according to the embodiment of the present invention by executing the programs in ROM 602 and / or RAM 603. It should be noted that the program can also be stored in one or more memories other than ROM 602 and RAM 603. The processor 601 can also perform various operations of the method flow according to the embodiment of the present invention by executing the programs stored in the one or more memories.

[0192] According to an embodiment of the present invention, the electronic device may further include an input / output (I / O) interface 605, which is also connected to the bus 604. The electronic device may further include one or more of the following components connected to the input / output (I / O) interface 605: an input portion 606 including a keyboard, a mouse, etc.; an output portion 607 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 608 including a hard disk, etc.; and a communication portion 609 including a network interface card such as a LAN card, a modem, etc. The communication portion 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the input / output (I / O) interface 605 as needed. A removable medium 611, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 610 as needed, so that a computer program read therefrom is installed into the storage portion 608 as needed.

[0193] According to an embodiment of the present invention, the method flow according to an embodiment of the present invention can be implemented as a computer software program. For example, an embodiment of the present invention includes a computer program product, which includes a computer program carried on a computer-readable storage medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 609, and / or installed from the removable medium 611. When the computer program is executed by the processor 601, the above-mentioned functions defined in the system of the embodiment of the present invention are executed. According to an embodiment of the present invention, the system, equipment, device, module, unit, etc. described above can be implemented by a computer program module.

[0194] The present invention also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiment; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present invention is implemented.

[0195] According to an embodiment of the present invention, the computer-readable storage medium may be a non-volatile computer-readable storage medium. For example, it may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, an apparatus or a device.

[0196] For example, according to an embodiment of the present invention, the computer-readable storage medium may include the ROM 602 and / or the RAM 603 described above and / or one or more memories other than the ROM 602 and the RAM 603 .

[0197] An embodiment of the present invention also includes a computer program product, which includes a computer program, which contains program code for executing the method provided by the embodiment of the present invention. When the computer program product runs on an electronic device, the program code is used to enable the electronic device to implement the transformer internal health status monitoring method based on the internal inspection robot provided by the embodiment of the present invention.

[0198] When the computer program is executed by the processor 601, the above functions defined in the system / device of the embodiment of the present invention are executed. According to the embodiment of the present invention, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0199] In one embodiment, the computer program may rely on tangible storage media such as optical storage devices, magnetic storage devices, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of signals on a network medium, and downloaded and installed through the communication part 609, and / or installed from a removable medium 611. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0200] According to an embodiment of the present invention, the program code for executing the computer program provided by the embodiment of the present invention can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (e.g., using an Internet service provider to connect through the Internet).

[0201] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present invention. In this regard, each box in the flowchart or block diagram may represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box may also occur in an order different from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions. It can be understood by those skilled in the art that the features recorded in the various embodiments of the present invention can be combined and / or combined in various ways, even if such a combination or combination is not explicitly recorded in the present invention. In particular, without departing from the spirit and teachings of the present invention, the features recorded in the various embodiments of the present invention can be combined and / or combined in various ways. All these combinations and / or combinations fall within the scope of the present invention.

[0202] The embodiments of the present invention are described above. However, these embodiments are only for the purpose of illustration, and are not intended to limit the scope of the present invention. Although each embodiment is described above, it does not mean that the measures in each embodiment cannot be used in combination advantageously. Without departing from the scope of the present invention, those skilled in the art may make various substitutions and modifications, which should all fall within the scope of the present invention.

Claims

1. A transformer internal health status monitoring method based on an internal inspection robot, characterized in that: The transformer internal health status monitoring method comprises: In response to a transformer health status monitoring request, acquiring a plurality of initial images captured by an internal inspection robot of the transformer; Perform feature matching based on the multiple initial images to obtain key point matching results between the multiple initial images; Performing depth alignment and splicing on the multiple initial images based on key point matching results between the multiple initial images to obtain a first spliced ​​image; Performing rectangular correction on the first stitched image to obtain a second stitched image; A health status detection is performed based on the second spliced ​​image to obtain an internal health status monitoring result of the transformer.

2. The transformer internal health status monitoring method according to claim 1, characterized in that: The performing feature matching based on the multiple initial images to obtain key point matching results between the multiple initial images includes: For each of the initial images, extract a key point heat map, a dense feature description map, and a reliability heat map from the initial image; Based on the reliability heat map of the initial image, determining a plurality of candidate key points from the key point heat map of the initial image; Determine, from the dense feature description graph of the initial image, first local features corresponding to each of the plurality of candidate key points; Feature matching is performed based on the multiple first local features of each of the multiple initial images to obtain key point matching results between the multiple initial images.

3. The transformer internal health status monitoring method according to claim 2, characterized in that: The performing feature matching based on the respective first local features of the plurality of initial images to obtain key point matching results between the plurality of initial images includes: For each of the initial images, interpolation processing is performed on a plurality of first local features of the initial image to obtain a plurality of second local features; Using a mutual nearest neighbor search method, matching the plurality of second local features of the plurality of initial images to obtain an initial matching result; Based on the initial matching results, dense matching is performed on the plurality of second local features of each of the plurality of initial images to obtain key point matching results between the plurality of initial images.

4. The transformer internal health status monitoring method according to claim 1, characterized in that: The step of performing depth alignment and stitching on the multiple initial images based on the key point matching results between the multiple initial images to obtain a first stitched image includes: determining a reference image from the plurality of initial images; For each of the other initial images among a plurality of other initial images, determining an association relationship between the other initial image and the reference image, the other initial images including initial images among the plurality of initial images except the reference image; Based on the association relationship between the other initial images and the reference image and the key point matching results between the multiple initial images, splicing the other initial images into the reference image space where the reference image is located; The multiple other initial images are traversed and processed to obtain the first stitched image.

5. The method for monitoring the internal health status of a transformer according to claim 4, characterized in that: The step of splicing the other initial images into the reference image space where the reference image is located based on the association relationship between the other initial images and the reference image and the key point matching results between the multiple initial images includes: In a case where the association relationship between the other initial image and the reference image is represented by the presence of an overlapping area between the other initial image and the reference image, determining a first affine transformation matrix and a second affine transformation matrix based on a plurality of first key points included in the other initial image and a plurality of second key points included in the reference image; Based on the first affine transformation matrix and the second affine transformation matrix, mapping the pixel points in the other initial images located in the overlapping area to the reference image space to obtain a first image area; Performing isomorphic transformation on the pixel points in the other initial images that are not located in the overlapping area to obtain a second image area located in the reference image space; The first image area, the second image area and the reference image are spliced ​​to splice the other initial images into a reference image space where the reference image is located.

6. The method for monitoring the internal health status of a transformer according to claim 5, characterized in that: The determining of a first affine transformation matrix and a second affine transformation matrix based on a plurality of first key points included in the other initial image and a plurality of second key points included in the reference image comprises: Determine a first target key point and a second target key point from the plurality of first key points and the plurality of second key points based on an average value of distances between each key point in the plurality of first key points and the plurality of second key points and a plurality of other key points, wherein the other key points include key points in the plurality of first key points and the plurality of second key points other than the above-mentioned key points; Taking the first target key point and the second target key point as cluster centers, clustering the plurality of first key points and the plurality of second key points to obtain a first initial clustering cluster and a second initial clustering cluster; A random sample consensus calculation is performed based on the first initial clustering cluster and the second initial clustering cluster respectively to obtain the first affine transformation matrix and the second affine transformation matrix.

7. The method for monitoring the internal health status of a transformer according to claim 6, characterized in that: The step of mapping the pixel points in the other initial images located in the overlapping area to the reference image space based on the first affine transformation matrix and the second affine transformation matrix to obtain a first image area includes: Eliminating outliers from the first initial cluster based on the first affine transformation matrix to obtain a first cluster; Eliminate outliers from the second initial cluster based on the second affine transformation matrix to obtain a second cluster; For each pixel point in the other initial image located in the overlapping area, determining a mapping weight of the pixel point based on a first distance from the pixel point to the first cluster and a second distance from the pixel point to the second cluster; Based on the mapping weights of the pixels, the first affine transformation matrix and the second affine transformation matrix, the pixels are mapped to the reference image space to obtain the first image area.

8. The method for monitoring the internal health status of a transformer according to claim 4, characterized in that: The step of splicing the other initial images into the reference image space where the reference image is located based on the association relationship between the other initial images and the reference image and the key point matching results between the multiple initial images includes: In a case where the association relationship between the other initial images and the reference image is represented by the other initial images not overlapping with the reference image, and there are n intermediate images between the other initial images and the reference image, sequentially mapping the overlapping area between the other initial images and the first intermediate image to the image space where the n intermediate images are located, until the overlapping area between the other initial images and the first intermediate image is mapped to the third image area in the reference image space, wherein there is an overlapping area between the nth intermediate image and the reference image; Performing isomorphic transformation on pixel points of non-overlapping areas between the other initial images and the first intermediate image, so as to sequentially map the non-overlapping areas between the other initial images and the first intermediate image to the image spaces where the n intermediate images are located, until the non-overlapping areas between the other initial images and the first intermediate image are mapped to a fourth image area in the reference image space; The third image area, the fourth image area and the reference image are spliced ​​to splice the other initial images into a reference image space where the reference image is located.

9. The method for monitoring the internal health status of a transformer according to claim 1, characterized in that: The performing rectangular correction on the first stitched image to obtain a second stitched image includes: Adding noise to the initial motion field data to obtain a noisy motion field; Using the first spliced ​​image and the image mask as constraints, denoising the noisy motion field to obtain a restored motion field; Performing deformation processing on the first stitched image based on the restored motion field to obtain a rectangular third stitched image; and Based on the confidence mask, the consistency constraint and the authenticity constraint, weighted sampling is performed on the third stitched image to obtain the second stitched image.

10. A transformer internal health status monitoring device based on an internal inspection robot, characterized in that: The transformer internal health status monitoring device comprises: An acquisition module, configured to acquire a plurality of initial images captured by an internal inspection robot of the transformer in response to a transformer health status monitoring request; A matching module, used for performing feature matching based on the multiple initial images to obtain key point matching results between the multiple initial images; A stitching module, configured to perform depth alignment and stitching on the multiple initial images based on key point matching results between the multiple initial images to obtain a first stitched image; A correction module, used for performing rectangular correction on the first stitched image to obtain a second stitched image; A detection module is used to perform health status detection based on the second spliced ​​image to obtain an internal health status monitoring result of the transformer.

Images