Electric power operation and maintenance scene inspection image splicing method

Abstract

The invention discloses an electric power operation and maintenance scene inspection image splicing method, and relates to the field of electric power operation and maintenance. The model framework comprises a coarse image alignment stage and an image fusion stage; in the coarse image alignment stage, two high-resolution operation and maintenance images with a certain overlapping ratio are used as input, the homography matrixes of the two high-resolution operation and maintenance images are estimated by using a depth homography network, and then the input images are distorted by using a splicing domain change method so that the input images are subjected to coarse alignment; 2, in a coarse image alignment stage network, a target image and a reference image are sequentially processed by a Focus layer and two Conv layers, a C3 layer is adopted after each Conv layer, the number of output channels of the Focus layer is 64, the number of filters of the two Conv layers is 128 and 256 respectively, and the network extracts features of the three layers; 3, an image fusion stage; 4, designing a loss function; compared with a traditional algorithm, the algorithm provided by the invention is more stable in performance in a splicing task of images shot in various environments. And the requirements of operation and maintenance scenes are met.

Description

Technical Field

[0001] The present invention relates to the technical field of electric power operation and maintenance, and in particular to a method for stitching inspection images of electric power operation and maintenance scenes. Background Art

[0002] In recent years, with the rapid development of the national economy, the requirements for safe power supply have been continuously improved, and the overall structure of the power grid has become increasingly large and complex. Power grid safety and equipment operation and maintenance monitoring have become the focus of power grid enterprises' unremitting efforts in production safety. From production to consumption, electric energy must go through multiple links such as power generation, transmission, transformation, and distribution. Any problem in any link will affect the normal supply of electric energy. When power equipment is operating in the field, it is often subject to the hazards of lightning strikes, acid rain corrosion, storms and dust, etc., resulting in problems such as broken strands in high-voltage wires and deterioration of insulators. If not repaired in time, it is very easy to cause power line disconnection or short circuit. Therefore, it is necessary to ensure the normal operation of the power grid system through continuous inspection and maintenance.

[0003] Electric power inspections have always been done manually. Inspection work is boring and tedious. A typical substation inspection involves many inspection points, which usually requires staff to spend 6-7 working hours to complete. In addition, each inspection is also a great burden on the physical strength of the operation and maintenance personnel. In addition, with the large-scale construction of power grids, the transmission, transformation and distribution equipment has increased significantly, but there is a serious shortage of professional technicians responsible for regular maintenance of new equipment. With the existing staff size, how to reduce manpower investment and improve operation and maintenance efficiency is a difficult problem that needs to be solved urgently by all power people.

[0004] With the innovative breakthroughs in digital information technologies such as big data, cloud computing, artificial intelligence, 5G communications, information security protection, and the Internet of Things, the digital economy, characterized by digital drive and based on data resources, has flourished. The new intelligent production method is accelerating, and drone-based inspection technology has been widely used in power operation and maintenance scenarios. However, the images provided by drones only contain part of the power equipment information, and the power operation and maintenance scenarios need to include wide-angle images of power equipment. At present, drone inspections cannot meet the needs of obtaining wide-angle images.

[0005] To this end, we provide a method for stitching inspection images in power operation and maintenance scenarios to solve the above problems. Summary of the invention

[0006] In response to the problems existing in the above-mentioned prior art, the present invention provides a method for stitching inspection images in power operation and maintenance scenarios. By using image stitching technology, operation and maintenance personnel can obtain wide-angle images of drone inspections and have a more comprehensive understanding of power system equipment and the surrounding environment. Moreover, the use of image stitching technology for remote operation and maintenance not only saves the time of field inspections by operation and maintenance personnel, making the operation and maintenance process more efficient and effective, but also improves the safety of the operation and maintenance process. In addition, combined with artificial intelligence algorithms, it can provide predictive maintenance suggestions and realize intelligent power operation and maintenance management.

[0007] In order to achieve the above-mentioned purpose, the present invention adopts a method for stitching inspection images of power operation and maintenance scenes, and the method for stitching inspection images of power operation and maintenance scenes comprises:

[0008] Step 1: Component model framework;

[0009] The model framework includes a coarse image alignment stage and an image fusion stage;

[0010] In the coarse image alignment stage, two high-resolution images with a certain overlap ratio are used as input, and the homography matrix of the two images is estimated using a deep homography network. Then, the input images are distorted using the splicing domain change method to achieve coarse alignment.

[0011] Among them, in the image fusion stage, the fusion network contains a low-resolution warping branch and a high-resolution refinement branch. The former learns the warping rules of image splicing at low resolution, and the latter is used to further improve the resolution of the output image of the low-resolution branch;

[0012] Step 2: In the coarse image alignment network, the target image and the reference image are processed by the Focus layer and two Conv layers in turn. The C3 layer is used after each Conv layer. The number of output channels of the Focus layer is 64, and the number of filters of the two Conv layers is 128 and 256 respectively. The network extracts three levels of features.

[0013] The feature correlation layer is used to calculate the global correlation and local correlation of the reference image features and the target image features;

[0014] The entire image is used as input, including all overlapping areas. When aligning the images, the invalid pixels in the distorted image are no longer filled, and the content corresponding to the invalid pixels is eliminated. At this time, the objective function is:

[0015]

[0016] Among them, ⊙ is the pixel-by-pixel multiplication, E is the sum I A All-one matrices of the same size, I A , I Bare the complete images of the reference image and the target image respectively, Operations to align an image to another image using an estimated homography;

[0017] Step 3: Image fusion stage;

[0018] In the image fusion stage, a fusion method from features to pixels is used; first, the deformation law of the image is learned in the low-resolution branch, and then it is refined in the high-resolution branch to improve the image quality;

[0019] In the low-resolution branch, the distorted image obtained from the alignment stage is first downsampled to obtain a low-resolution image of size 256×256. Then, the image is reconstructed from semantic features to RGB pixel values ​​in the fusion network, and finally the low-resolution image fusion result is output.

[0020] In the high-resolution branch, the low-resolution fusion result and the distorted image are fused together as input, the image features are further processed and refined in the fusion network, and finally a high-resolution RGB image is output;

[0021] Step 4: Design loss function;

[0022] When designing the loss function, content mask and seam mask are introduced;

[0023] Among them, the content mask is used to extract the content belonging to the reference image and the warped target image in the output fused image;

[0024] The seam refers to the intersection boundary between the reference image and the warped target image, and the seam mask is used to extract the seam content in the output fused image, the reference image, and the warped target image.

[0025] As a further optimization of the above scheme, in the second step: after extracting multi-scale features and calculating feature correlations, a regression network is used to predict the coordinate offsets corresponding to the homography transformation of the four vertices. The offsets are fed into the DLT function to calculate the corresponding homography matrix. Each layer uses the features to estimate the homography, and the estimated matrix is ​​transmitted to the next layer.

[0026] As a further optimization of the above scheme, in the first step, the stitching domain transformation method is used when the image is distorted, and the minimum rectangular boundary of the stitching image is specified as the stitching domain. In the stitching domain transformation method, first, the positions of the four vertices of the deformed target image are calculated:

[0027]

[0028] in, and are the kth vertices of the deformed target image and the original target image, (Δx k,Δy k ) is the offset of the deformed image relative to the original image, and then the size of the deformed image (W) can be obtained through the formula * ×H * );

[0029]

[0030] in is the vertex coordinate of the reference image, and its value is same;

[0031] Finally, the warping of the target image is completed in the stitching domain;

[0032]

[0033] Among them, I and H are the unit matrix and the homography matrix obtained by homography estimation of the target image respectively; It is the operation of distorting the image using a 3×3 matrix; (I A ,I B ) is the input image, (I AW ,I BW ) is a distorted image.

[0034] As a further optimization of the above scheme, the number of filters in each convolutional layer in the low-resolution network in the third step is 64, 64, 128, 128, 256, 256, 512, 512, 256, 256, 128, 128, 64, 64, 3, and jump connections are used to connect low-level features and high-level features at the same resolution.

[0035] As a further optimization of the above scheme, the high-resolution branch in the third step includes three independent convolutional layers and 8 residual blocks. The number of convolutional layer filters is 64, 64, and 3. Each residual block consists of convolution, relu, convolution, sum, and relu, and retains the detailed information of the input image by merging the features before and after the residual.

[0036] As a further optimization of the above scheme, in the fourth step, by replacing I in formula (5) A , I B By replacing the matrix E with the same size, the content mask can be obtained, and the seam mask can be further obtained using the content mask:

[0037]

[0038] Among them, M AC , M BC is the content mask, M AS , M BSis the seam mask, (ij) is the coordinate of the pixel in the image, * is the convolution operation, Limit all elements to values ​​between 0 and 1;

[0039] Based on the content mask and seam mask, design the content loss function and seam loss function The content loss function is used to bring the features of the fused image closer to those of the reference image and the distorted target image, while the seam loss function brings the content of the fused image at the seam closer to that of the reference image and the distorted target image.

[0040]

[0041] Among them, S LR is the fused image obtained at low resolution. L1 loss and perceptual loss are calculated separately; the total loss function in the low-resolution case is:

[0042]

[0043] Among them, λ C ,λ s are the weights of content mask loss and seam mask loss respectively;

[0044] The fused image output by the high-resolution branch has similar content loss function and seam loss function, and the total loss function of this branch is:

[0045]

[0046] in, is the content loss function for high-resolution cases, is the seam loss function for high resolution.

[0047] As a further optimization of the above scheme, a joint loss function of the two branches is designed to bring the output results of the high-resolution branch and the low-resolution branch closer:

[0048]

[0049] in, It is the image obtained after resizing the high-resolution image to 256×256;

[0050] After comprehensively considering all loss functions, the total loss function of the network in the fusion stage is as follows:

[0051]

[0052] Among them, ω LR ,ω HR ,ωCS is the weight corresponding to each loss function.

[0053] The power operation and maintenance scene inspection image stitching method of the present invention has the following beneficial effects:

[0054] The method for stitching inspection images in power operation and maintenance scenarios of the present invention has a more stable performance than traditional algorithms in stitching images taken in various environments and is more suitable for the needs of operation and maintenance scenarios.

[0055] With reference to the following description and drawings, a specific embodiment of the present invention is disclosed in detail, indicating the manner in which the principles of the present invention can be adopted. It should be understood that the scope of the embodiments of the present invention is not limited thereby, and within the spirit and scope of the appended claims, the embodiments of the present invention include many changes, modifications and equivalents. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 This is a flow chart of the inspection image stitching method for power operation and maintenance scenarios of the present invention. DETAILED DESCRIPTION

[0057] Electricity is an indispensable part of people's lives and production today. With the rapid development of my country's economy, the scale of my country's power grid has become increasingly large, which has posed new challenges to the power operation and maintenance system. The traditional manual inspection method has problems such as high operation and maintenance pressure and shortage of operation and maintenance personnel. Although the inspection method based on drones can solve the above problems to a certain extent, the shooting field of drones is limited and cannot provide images with a wide perspective. With the rapid development of the field of computer vision, more and more scholars are considering using image stitching technology to solve this limitation. Based on a comprehensive analysis of the research results and current status in the field of image stitching, the present invention conducts an in-depth study on the stitching of operation and maintenance scene images.

[0058] Firstly, the traditional feature-based image stitching method and deep learning-based method are analyzed, and it is determined to use the deep learning-based image stitching algorithm for operation and maintenance image stitching.

[0059] Then, we conducted in-depth research on the objectives of each stage in the image stitching task and the relevant knowledge of deep learning technology, and based on this, determined the overall structure of the unsupervised operation and maintenance image stitching network based on deep learning.

[0060] Finally, the effectiveness and correctness of the image stitching network framework proposed in this paper were verified through experiments, proving that the inspection image stitching method for power operation and maintenance scenarios proposed in this invention has practical application value in power operation and maintenance scenarios.

[0061] Please refer to the instruction manual Figure 1The present invention provides a technical solution: a method for stitching inspection images in power operation and maintenance scenarios.

[0062] Implementation process:

[0063] Step 1: Model framework;

[0064] The inspection image stitching method for power operation and maintenance scenarios proposed in the present invention includes two stages: coarse image alignment and image fusion.

[0065] Among them, in the coarse image alignment stage, two high-resolution operation images with a certain overlap rate are taken as input, and the deep homography network is used to estimate the homography matrix of the two, and then the splicing domain change method is used to distort the input images to make them coarsely aligned.

[0066] In the image fusion stage, the fusion network contains a low-resolution warping branch and a high-resolution refinement branch. The former learns the warping rules for image stitching at low resolution, and the latter is used to further improve the resolution of the output image of the low-resolution branch.

[0067] Step 2: In the network of the coarse image alignment stage, the target image and the reference image are processed by the Focus layer and two Conv layers in turn, and the C3 layer is used after each Conv layer. The number of output channels of the Focus layer is 64, and the number of filters of the two Conv layers is 128 and 256 respectively. The network extracts three levels of features. In order to improve the accuracy of homography estimation under large baseline conditions, the present invention uses a feature correlation layer to calculate the global correlation and local correlation of reference image features and target image features. After extracting multi-scale features and calculating feature correlation, a regression network is used to predict the coordinate offsets corresponding to the homography transformation of the four vertices. The offset is sent to the DLT function to calculate the corresponding homography matrix. Each layer uses features to perform homography estimation, and transmits the estimated matrix to the lower layer to improve the accuracy of homography estimation.

[0068] Existing deep homography networks all take partial images as input, and the objective functions of these methods can be expressed as:

[0069]

[0070] Among them, I A , I B are the complete images of the reference image and the target image respectively, is a function to extract a partial area from a complete image, An operation that aligns an image to another image using an estimated homography.

[0071] From formula (1), we can see that in order to make the distorted image aligned with the reference image as much as possible, we can use the content around the selected area to fill the invalid pixels in the overlapping area of ​​the distorted image. However, this method works better when the baseline is small or medium (that is, the shooting angle of the two stitched images is not large), but it works poorly in large baseline scenes, especially when the overlapping area of ​​the two images is extremely small.

[0072] To solve this problem, the present invention adopts a new strategy, that is, the entire image is used as input, including all overlapping areas, and when aligning images, the invalid pixels in the distorted image are no longer filled, but the content corresponding to the invalid pixels is ablated. In this case, the objective function is:

[0073]

[0074] Among them, ⊙ is the pixel-by-pixel multiplication, E is the sum I A All-one matrices of the same size.

[0075] The overlap rates of stitched images are different, resulting in different resolutions of the final images. Therefore, the commonly used image distortion method is to set a fixed high-resolution output space and perform image transformation on this basis, which can effectively ensure that the image content remains in the output space. However, this method will waste most of the space and impose a heavy burden on the GPU memory. Therefore, the present invention proposes a stitching domain transformation method, which stipulates that the minimum rectangular boundary of the stitched image is the stitching domain. This method can save space, reduce memory pressure, and ensure the integrity of the output image content.

[0076] In the splicing domain transformation method, first, the positions of the four vertices of the deformed target image are calculated:

[0077]

[0078] in, and are the kth vertices of the deformed target image and the original target image, (Δx k ,Δy k ) is the offset of the deformed image relative to the original image. Then, the size of the deformed image (W) can be obtained by the following formula: * ×H * ).

[0079]

[0080] in is the vertex coordinate of the reference image, and its value is same.

[0081] Finally, the warping of the target image is done in the stitching domain.

[0082]

[0083] Among them, I and H are the unit matrix and the homography matrix obtained by estimating the homography of the target image, respectively. It is an operation to distort the image using a 3×3 matrix. A ,I B ) is the input image, (I AW ,I BW ) is a distorted image.

[0084] Step 3: Image fusion stage;

[0085] In the image fusion stage, considering the limitation that a single homography matrix can only represent spatial transformations of the same depth, after the first stage, the input images are not completely aligned in the real data set, and there are still slight misalignments, resulting in artifacts in the final spliced ​​image. In order to break the bottleneck of a single homography and further improve the alignment accuracy, the present invention proposes a fusion method from features to pixels.

[0086] In the case of high resolution, as the resolution increases, the receptive field decreases relatively and cannot adapt to the situation of large baseline or low overlap rate. In order to enable the receptive field of the image to fully perceive the unaligned area and output a high-quality stitched image, the present invention proposes a fusion stage network combining a low-resolution branch with a high-resolution branch. First, the deformation law of the image is learned in the low-resolution branch, and then it is refined in the high-resolution branch to improve the image quality.

[0087] In the low-resolution branch, the warped image obtained from the alignment stage is first downsampled to obtain a low-resolution image of size 256×256, and then the image is reconstructed from semantic features to RGB pixel values ​​in the fusion network, and finally the low-resolution image fusion result is output. The number of filters in each convolution layer in the low-resolution network is 64, 64, 128, 128, 256, 256, 512, 512, 256, 256, 128, 128, 64, 64, 3, and the low-level features and high-level features at the same resolution are linked through jump connections. In the high-resolution branch, the low-resolution fusion result and the warped image are fused together as input, and the image features are further processed and refined in the fusion network, and finally the high-resolution RGB image is output. The high-resolution branch includes three independent convolutional layers and 8 residual blocks, and the number of convolutional layer filters is 64, 64, 3. Each residual block consists of convolution, relu, convolution, sum, relu. And by merging the features before and after the residual, the detailed information of the input image is retained.

[0088] Step 4: Loss function;

[0089] When designing the loss function, the present invention introduces a content mask and a seam mask, wherein the content mask is used to extract the content belonging to the reference image and the distorted target image in the output fused image, the seam refers to the intersection boundary of the reference image and the distorted target image, and the seam mask is used to extract the seam content in the output fused image, the reference image, and the distorted target image. Specifically, by replacing I in formula (6) with A , I B By replacing it with an all-one matrix E of the same size, the content mask can be obtained, and the seam mask can be further obtained using the content mask, as shown in the following formula.

[0090]

[0091] Among them, M AC , M BC is the content mask, M AS , M BS is the seam mask, (ij) is the coordinate of the pixel in the image, * is the convolution operation, Clamp all elements to values ​​between 0 and 1.

[0092] Based on the content mask and the seam mask, the present invention also designs a content loss function and seam loss function The content loss function is used to bring the features of the fused image closer to those of the reference image and the distorted target image, while the seam loss function brings the content of the fused image at the seam closer to the content of the reference image and the distorted target image.

[0093]

[0094] Among them, S LR is the fused image obtained at low resolution. L1 loss and perceptual loss are calculated separately. The total loss function in the low-resolution case is:

[0095]

[0096] Among them, λ C ,λ s are the weights of content mask loss and seam mask loss, respectively.

[0097] The fused image output by the high-resolution branch has similar content loss function and seam loss function, and the total loss function of this branch is:

[0098]

[0099] in, is the content loss function for high-resolution cases, is the seam loss function for high resolution.

[0100] Since the receptive field of the network is relatively small under high resolution, parallax artifacts are prone to appear in the reconstruction results. In order to improve the resolution and eliminate the parallax artifacts, the present invention designs a joint loss function of the two branches to bring the output results of the high-resolution branch and the low-resolution branch closer:

[0101]

[0102] in, It is the image obtained after resizing the high-resolution image to 256×256.

[0103] After comprehensively considering all loss functions, the total loss function of the network in the fusion stage is as follows:

[0104]

[0105] Among them, ω LR ,ω HR ,ω CS is the weight corresponding to each loss function.

[0106] Experimental Results

[0107] 1. Homography Estimation Comparison

[0108] This paper compares the deep learning-based unsupervised image stitching framework with traditional image stitching algorithms such as SIFT+RANSAC, ORB+RANSAC, and APAP[9] on synthetic and real datasets.

[0109] The experiment is divided into the alignment phase experiment and the fusion phase experiment. In the alignment phase, on Warped MS-COCO, the model converges after 100 epochs, and on the real dataset, the model converges after 40 epochs.

[0110] The alignment effects of various algorithms on the input images are compared, and the quantitative experimental results are shown in Tables 1 and 2.

[0111] Table 1 Quantitative comparison of algorithms on Warped MS-COCO

[0112]

[0113] Table 2 Quantitative comparison of algorithms on UAV dataset

[0114]

[0115] By analyzing the results, we can easily draw the following conclusion: the algorithm adopted by the present invention can achieve better alignment effect whether on synthetic data sets or real data sets.

[0116] 2. Experimental results and analysis

[0117] In the fusion stage, the fusion results of the algorithm of the present invention are compared with those of SIFT+RANSAC and APAP algorithms.

[0118] The stitching results of the SIFT+RANSAC algorithm have obvious differences at the seams, and the light and dark differences in some parts are obvious. This is because the SIFT algorithm is insensitive to light changes. Although the APAP algorithm has no obvious changes at the seams, obvious parallax artifacts appear in many places in the picture, which may cause the operation and maintenance personnel to make wrong judgments in the operation and maintenance scenario. In the fusion stage, the algorithm of the present invention adopts the idea of ​​reconstruction from semantic features to RGB pixel values, which effectively avoids the appearance of parallax artifacts. And because the loss function related to the content mask and the seam mask is introduced at this stage, the image seams are smoother, there is no obvious difference on both sides of the seams, the details of the power equipment are obvious, and there is no obvious distortion, which greatly meets the needs of power operation and maintenance in terms of image quality.

[0119] In terms of the speed of image restoration, the average speed of stitching images using SIFT+RANSAC is 2.4 seconds per group, and APAP takes about 3.7 seconds to stitch each group of images. Our model took 31 seconds to stitch 1,000 groups of images in the test set, with an average of 0.03 seconds per group. This proves that the performance of the model of the present invention is better than that of traditional algorithms when processing large amounts of data.

[0120] From the perspective of algorithm robustness, the algorithm of the present invention has good stitching effect on images taken in different environments and with different light and dark levels. However, the performance of feature-based stitching methods is easily affected by the number and distribution of feature points, and the robustness is weak in different scenes. The present invention counts the number of stitching failures during the stitching process, and the results are shown in Table 3.

[0121] Table 3 Comparison of robustness of splicing methods

[0122]

[0123] From the data in the table, we can observe that the algorithm of the present invention has more stable performance in image stitching tasks taken in various environments than the traditional algorithm, and is more suitable for the needs of operation and maintenance scenarios.

Claims

1. A method for stitching inspection images in power operation and maintenance scenarios, characterized in that: include: Step 1: Component model framework; The model framework includes a coarse image alignment stage and an image fusion stage; In the coarse image alignment stage, two high-resolution images with a certain overlap ratio are used as input, and the homography matrix of the two images is estimated using a deep homography network. Then, the input images are distorted using the splicing domain change method to achieve coarse alignment. Among them, in the image fusion stage, the fusion network contains a low-resolution warping branch and a high-resolution refinement branch. The former learns the warping rules of image splicing at low resolution, and the latter is used to further improve the resolution of the output image of the low-resolution branch; Step 2: In the coarse image alignment network, the target image and the reference image are processed by the Focus layer and two Conv layers in turn. The C3 layer is used after each Conv layer. The number of output channels of the Focus layer is 64, and the number of filters of the two Conv layers is 128 and 256 respectively. The network extracts three levels of features. The feature correlation layer is used to calculate the global correlation and local correlation of the reference image features and the target image features; The entire image is used as input, including all overlapping areas. When aligning the images, the invalid pixels in the distorted image are no longer filled, and the content corresponding to the invalid pixels is eliminated. At this time, the objective function is: Among them, ⊙ is the pixel-by-pixel multiplication, E is the sum I A All-one matrices of the same size, I A , I B are the complete images of the reference image and the target image respectively, Operations to align an image to another image using an estimated homography; Step 3: Image fusion stage; In the image fusion stage, a fusion method from features to pixels is used; first, the deformation law of the image is learned in the low-resolution branch, and then it is refined in the high-resolution branch to improve the image quality; In the low-resolution branch, the distorted image obtained from the alignment stage is first downsampled to obtain a low-resolution image of size 256×256. Then, the image is reconstructed from semantic features to RGB pixel values ​​in the fusion network, and finally the low-resolution image fusion result is output. In the high-resolution branch, the low-resolution fusion result and the distorted image are fused together as input, the image features are further processed and refined in the fusion network, and finally a high-resolution RGB image is output; Step 4: Design loss function; When designing the loss function, content mask and seam mask are introduced; Among them, the content mask is used to extract the content belonging to the reference image and the warped target image in the output fused image; The seam refers to the intersection boundary between the reference image and the warped target image, and the seam mask is used to extract the seam content in the output fused image, the reference image, and the warped target image.

2. The method for stitching inspection images of power operation and maintenance scenes according to claim 1 is characterized by: In the second step: after extracting multi-scale features and calculating feature correlations, a regression network is used to predict the coordinate offsets corresponding to the homography transformation of the four vertices. The offsets are fed into the DLT function to calculate the corresponding homography matrix. Each layer uses the features to estimate the homography and transmits the estimated matrix to the next layer.

3. The method for stitching inspection images of power operation and maintenance scenes according to claim 1 is characterized by: In the first step, the stitching domain transformation method is used when the image is distorted. The minimum rectangular boundary of the stitching image is specified as the stitching domain. In the stitching domain transformation method, first, the positions of the four vertices of the deformed target image are calculated: in, and are the kth vertices of the deformed target image and the original target image, (Δx k ,Δy k ) is the offset of the deformed image relative to the original image, and then the size of the deformed image (W) can be obtained through the formula * ×H * ); in is the vertex coordinate of the reference image, and its value is same; Finally, the warping of the target image is completed in the stitching domain; Among them, I and H are the unit matrix and the homography matrix obtained by homography estimation of the target image respectively; It is the operation of distorting the image using a 3×3 matrix; (I A ,I B ) is the input image, (I AW ,I BW ) is a distorted image.

4. The method for stitching inspection images of power operation and maintenance scenes according to claim 1 is characterized by: In the third step, the number of filters in each convolutional layer of the low-resolution network is 64, 64, 128, 128, 256, 256, 512, 512, 256, 256, 128, 128, 64, 64, 3, and jump connections are used to connect the low-level features and high-level features at the same resolution.

5. The method for stitching inspection images of power operation and maintenance scenes according to claim 1 is characterized by: In the third step, the high-resolution branch includes three independent convolutional layers and 8 residual blocks. The number of convolutional layer filters is 64, 64, and 3. Each residual block consists of convolution, relu, convolution, sum, and relu, and retains the detailed information of the input image by merging the features before and after the residual.

6. The method for stitching inspection images of power operation and maintenance scenes according to claim 1 is characterized by: In the fourth step, by replacing I in formula (5) A , I B By replacing the matrix E with the same size, the content mask can be obtained, and the seam mask can be further obtained using the content mask: Among them, M AC , M BC is the content mask, M AS , M BS is the seam mask, (ij) is the coordinate of the pixel in the image, * is the convolution operation, Limit all elements to values ​​between 0 and 1; Based on the content mask and seam mask, design the content loss function and seam loss function The content loss function is used to bring the features of the fused image closer to those of the reference image and the distorted target image, while the seam loss function brings the content of the fused image at the seam closer to that of the reference image and the distorted target image. Among them, S LR is the fused image obtained at low resolution. L1 loss and perceptual loss are calculated separately; the total loss function in the low-resolution case is: Among them, λ C ,λ s are the weights of content mask loss and seam mask loss respectively; The fused image output by the high-resolution branch has similar content loss function and seam loss function, and the total loss function of this branch is: in, is the content loss function for high-resolution cases, is the seam loss function for high resolution.

7. The method for stitching inspection images of power operation and maintenance scenes according to claim 1 is characterized by: It also includes the design of a joint loss function for the two branches, which is used to bring the output results of the high-resolution branch and the low-resolution branch closer: in, It is the image obtained after resizing the high-resolution image to 256×256; After comprehensively considering all loss functions, the total loss function of the network in the fusion stage is as follows: Among them, ω LR ,ω HR ,ω CS is the weight corresponding to each loss function.

Images