A power equipment image registration method based on deep learning

Abstract

The present application relates to a kind of power equipment image registration methods based on deep learning, including image feature extraction step, it includes visible light image feature extraction branch;Thermal imaging image feature extraction branch;Edge-labeled visible light image feature extraction branch;Cross attention mechanism step is applied: the first interaction information of the feature map of the thermal imaging image is passed to the feature map of visible light image and the second interaction information of the feature map of visible light image is passed to the feature map of thermal imaging image;Feature learning step is input into image matching learning network, obtains conversion matrix result with image feature map and interaction information.This method is by learning the correlation coefficient of different images, composition learning network is carried out feature identification, finally realizes the accurate registration of power equipment image.The registration of power equipment image is realized by the above method, with high registration accuracy, wide applicability and other advantages, can be widely applied in the field of power equipment safety monitoring.

Description

Technical Field

[0001] The present invention relates to the technical field of image registration, and in particular to a method for image registration of electric power equipment based on deep learning. Background Art

[0002] With the emergence of the smart grid concept and the widespread application of deep learning in artificial intelligence, intelligent safety monitoring of power systems is becoming a research hotspot and development direction in power research and engineering applications. In recent years, the power industry has increasingly favored unmanned monitoring of power equipment safety. The use of thermal imaging cameras to monitor power equipment in real time and then analyze and process the images to detect potential temperature anomalies has become a key component of intelligent power systems.

[0003] In scenarios where power grid equipment is unmanned, it is feasible to monitor power grid equipment using thermal imaging technology. However, thermal imaging images have low pixel counts. Although they can display temperature change information in the scene, the recognition of objects in the scene is limited by the low resolution, making it difficult to view imaging details. Visible light imaging technology is already very mature and can achieve higher image resolution. During maintenance, color images can be used to zoom in and observe equipment details in key areas. Therefore, for the needs of power grid anomaly monitoring and maintenance, it is necessary to provide a new monitoring scene display solution that integrates thermal imaging images and visible light images (black and white or color images), so that when the temperature is abnormal, the details of the abnormal area can be quickly viewed and obtained, guiding the determination of treatment plans for the abnormality. Summary of the Invention

[0004] In the smart grid sector, thermal imaging technology based on temperature imaging can detect temperature anomalies in grid equipment. However, its low pixel count makes it difficult to identify and locate the cause of heating when an anomaly occurs, allowing for rapid repairs. To address this issue, the present invention provides a deep learning-based image registration method for power equipment, achieving image registration and fusion, thereby addressing this technical issue.

[0005] The technical solution of the present invention provides a method for power equipment image registration based on deep learning, comprising the following steps:

[0006] S1, image feature extraction step, including the following feature extraction branches:

[0007] The visible light image feature extraction branch takes the visible light image as input and obtains the feature map of the visible light image after processing by the visible light image feature extraction network;

[0008] The thermal imaging image feature extraction branch takes the thermal imaging image as input and obtains the feature map of the thermal imaging image after processing by the thermal imaging image feature extraction network;

[0009] The edge-annotated visible light image feature extraction branch takes as input a visible light image with key edge regions annotated. After processing by the additional image feature extraction network, the image feature map with additional edge annotations is obtained.

[0010] S2. Applying a cross-attention mechanism: Applying a cross-attention module to obtain first interaction information between a feature map of the thermal image and a feature map of the visible light image, and second interaction information between a feature map of the visible light image and a feature map of the thermal image, respectively. Determine a correlation coefficient matrix between the thermal image and the visible light image based on the first interaction information and the second interaction information.

[0011] S3, feature learning step: input the image feature map obtained in the image feature extraction step S1 and the correlation coefficient matrix obtained in the cross-attention mechanism in S2 into the image matching learning network to obtain the conversion matrix result of the thermal imaging image into the visible light image coordinate system.

[0012] Preferably, in the step S2 of applying the cross attention mechanism, the calculation formula of the interaction information is:

[0013]

[0014] Among them, Q and K represent image features. d is QK T Dimensions,

[0015] The first mutual information is defined as m1=CrossAtt(f1,f2);

[0016] The second mutual information is defined as m2=CrossAtt(f2,f1);

[0017] Among them, f1 represents the feature map of the visible light image, and f2 represents the feature map of the thermal imaging image.

[0018] Preferably, in the step S2 of applying the cross-attention mechanism, the correlation coefficient matrix G2 of the thermal imaging image is a feature weighted matrix calculated by multiplying the feature map f2 of the thermal imaging image and the corresponding pixels of the second interactive information m2; the correlation coefficient matrix G1 of the visible light image is a feature weighted matrix calculated by multiplying the feature map f1 of the visible light image and the corresponding pixels of the first interactive information m1.

[0019] Preferably, assuming that the initial image feature block, the matched feature block and the unmatched feature block are a, p, and n, the loss function L of the image matching learning network is defined as:

[0020] L=βP an -γP ap

[0021] Among them, P ap =||f(a)-f(p)||2 and P an =||f(a)-f(n)||2 represents the Euclidean distance between two image feature descriptions.

[0022] Preferably, the loss function L of the image matching learning network is defined as:

[0023] L=βP an -γP ap +G

[0024] Among them, P ap =||f(a)-f(p)||2 and P an =||f(a)-f(n)||2 represents the Euclidean distance between two image feature descriptions; G is the edge judgment influencing factor.

[0025] Preferably, the edge judgment influencing factor G is:

[0026]

[0027] Where λ is the adjustment parameter.

[0028] This method extracts image features using different feature scales. It then uses the correlation coefficients of different images to jointly learn and form a learning network for feature recognition, ultimately achieving accurate registration of power equipment images. This method offers the advantages of high registration accuracy and wide applicability, and is widely applicable in the field of power equipment safety monitoring.

[0029] This invention relates to the field of image processing technology, and more particularly to a deep learning-based method for image registration of power equipment. This method incorporates a cross-attention module across image blocks, calculates information interaction between two points in different image blocks, and fuses the correlation coefficients between color and thermal images for similarity learning, ultimately achieving precise registration of color and thermal images. Compared to traditional methods, this method offers advantages such as high registration accuracy, strong robustness, and wide applicability, and can be widely used in power equipment safety monitoring. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 This is a flow chart of a method for image registration of power equipment based on deep learning according to the present invention;

[0031] Figure 2 Schematic diagram of the principle of optimizing the cross-attention mechanism of the present invention;

[0032] Figure 3 It is the alignment result when the power equipment image registration method based on deep learning of the present invention adopts edge annotation information. DETAILED DESCRIPTION

[0033] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. In this specification, the size ratios in the drawings do not represent the actual size ratios, but are only used to reflect the relative positional relationship and connection relationship between the various components. Components with the same name or the same number represent similar or identical structures and are only for illustrative purposes.

[0034] To solve the problem that thermal imaging images can only determine the temperature distribution of equipment, but the resolution is insufficient and it is difficult to observe the details of the equipment, the smart grid design can adopt the method of fusing thermal imaging images and visible light images. In actual installation scenarios, thermal imaging cameras and visible light cameras are generally not necessarily installed in the same place, so the image perspectives of the same equipment obtained by the two systems are often different. In addition, the complete sets of equipment for power equipment monitoring all use mature commercial solutions, among which the imaging resolution of color cameras has obvious advantages over the imaging resolution of thermal imaging images. In other words, the number of pixels of the two is very different. In view of the differences in shooting perspective and pixel size, it is difficult to directly fuse thermal imaging images with visible light images. To this end, the present invention provides a method for image registration of power equipment based on deep learning.

[0035] The image registration method of the present invention aims to determine the conversion matrix for aligning the thermal imaging image to the visible light image through deep learning based on the input thermal imaging image and the visible light image. After the present invention obtains the conversion matrix, the thermal imaging image is rotated, scaled, translated, and the like by applying the conversion matrix to the thermal imaging image, thereby aligning the thermal imaging image to the visible light image and finally fusing it into the visible light image. The deep learning model used in the following image registration method is pre-trained using paired image data of pre-aligned thermal imaging images and visible light images (i.e., the conversion matrix is ​​determined). The model training process based on a specific model according to the required samples belongs to the prior art and will not be described in detail in this application.

[0036] In the application process, Figure 1 、 2 As shown, the power equipment image registration method based on deep learning of the present application generally includes the following steps.

[0037] S1. Image feature extraction step. This step processes the input image through a feature extraction network to obtain a feature map corresponding to the input image. Specifically, it includes several branches that can be processed in parallel. These branches are:

[0038] Visible light image feature extraction branch: The input is a visible light image. After being processed by the visible light image feature extraction network, the feature map f1 of the visible light image is obtained.

[0039] Thermal imaging image feature extraction branch: The input is a thermal imaging image. After being processed by the thermal imaging image feature extraction network, the feature map f2 of the thermal imaging image is obtained.

[0040] Edge-annotated visible light image feature extraction branch: The input is a visible light image with key edge regions annotated. After processing by the additional image feature extraction network, the image feature map E with additional edge annotations is obtained.

[0041] S2. Apply the cross attention mechanism step.

[0042] To improve the model's accuracy in learning this task, a cross-attention mechanism is applied between the visible light image feature map f1 and the thermal image feature map f2 to obtain the correlation coefficient between them. Specifically, the cross-attention module obtains the first mutual information m1 transmitted from the thermal image feature map f2 to the visible light image feature map f1, and the second mutual information m2 transmitted from the visible light image feature map f1 to the thermal image feature map f2.

[0043] S3, feature learning step. The image feature map obtained in the image feature extraction step S1 and the correlation coefficient obtained by applying the cross-attention mechanism in S2 are input into the image matching learning network. The image matching learning network includes several convolutional layers and several pre-pooling layers. After processing, the deep learning result is obtained, that is, the conversion matrix result of converting the thermal imaging image into the visible light image coordinate system.

[0044] For the S1 image feature extraction step, the thermal and visible light images can be sourced from thermal and visible light cameras deployed on-site. Since the cameras output video streams exceeding 20 fps, the actual processing involves frame extraction from the video streams to obtain the thermal and visible light images at the same moment. In this solution, paired thermal and visible light images are obtained by randomly extracting 5 frames per second. To facilitate feature matching calculations during feature extraction and feature learning, the input paired thermal and visible light images must be processed. Generally, the image size and resolution must be adjusted to achieve a uniform input image size. Based on this, a pre-trained edge feature extraction module analyzes and extracts the contour edges in the visible light image, prioritizing edge information of power equipment while removing those that identify background patterns. This aims to increase the weight of feature learning for power equipment during feature extraction and learning by additionally annotating edge information of power equipment. This results in a higher accuracy model for image registration in power equipment scenarios, speeding up the model processing and improving processing efficiency.

[0045] Since the input images are uniformly processed to have the same size and resolution, to ensure that the output image feature maps have the same dimensionality, the feature extraction networks used preferably have the same network architecture. Of course, the specific weight parameters of the convolutional kernels within the convolutional layers are determined during the training process and therefore do not necessarily need to be the same. The input image and output feature map dimensions are 640*480 pixels. After a series of convolution operations, the small-scale feature description network produces a specific number of feature channels. When the output of the small-scale network is passed to the large-scale network, this channel number information is also passed along, and a larger receptive field is used (3*3 pixels -> 7*7 pixels). This allows neurons to capture a wider range of regions in the input feature map, helping to capture a wider range of contextual information and better extracting global structural information from the image.

[0046] This feature extraction network is configured with two descriptor networks of different feature scales (with receptive fields of 3*3 and 7*7 pixels, respectively). The small-scale feature descriptor network (3*3 pixels) is trained first, followed by the large-scale feature descriptor network (7*7 pixels). The output feature maps of the small-scale feature descriptor network serve as input to the large-scale feature descriptor network, yielding feature maps for different input images. Feature scale refers to the scale used to describe information within different spatial ranges within an image. Typically, feature scale can be achieved by applying filters of different sizes to a convolutional neural network. Filters of different sizes can capture features at different scales within an image, such as edges, textures, and the overall structure of objects. Different feature scales affect feature extraction and representation capabilities. Smaller feature scales are generally better suited to capturing local details in an image, such as textures and edges, while larger feature scales are better suited to capturing global structural information, such as the overall shape and layout of objects. Therefore, using different feature scales can provide a more comprehensive and diverse image representation, thereby improving feature richness and representation capabilities.

[0047] like Figure 3 As shown in the figure, the thermal imaging grayscale image can mainly identify the area where the equipment temperature is abnormal, but since the background temperature difference in the thermal imaging image is not large, the edge of the equipment cannot be clearly displayed on the image. In view of actual needs, we need to align the main body of the power equipment in the thermal imaging grayscale image with the main body of the power equipment in the color image, so as to align the equipment outline of the color image to the thermal imaging image and finally obtain the accurate equipment edge outline in the thermal imaging grayscale image (thermodynamic image). Thermal imaging grayscale images have the disadvantages of image blur, high noise and unclear equipment edges, such as Figure 3As shown in the figure, experiments show that conventional alignment results without edge annotations have large deviations, with edge contours significantly deviating from the device edges in the thermodynamic image, making it difficult to effectively apply in production environments. When a deep learning method with edge annotation is used, the edge alignment effect is significantly better than that without edge annotation information.

[0048] In the step S2 of applying the cross-attention mechanism, in order to improve the recognition of the correlation between the thermal imaging image and the visible light image, and thereby improve the recognition efficiency and accuracy of the image subject, the cross-attention mechanism is applied to the feature map f2 of the thermal imaging image and the feature map f1 of the visible light image obtained in the image feature extraction step S1.

[0049] Input f1 and f2 into the cross attention module, which is used to calculate the reorganized mapping of the feature vectors between any two points of f1 and f2, denoted as m1 and m2 respectively. m1 is the information transferred from f2 to f1, and m2 is the information transferred from f1 to f2. The specific calculation formula is as follows:

[0050]

[0051] Among them, Q and K represent image features. d is QK T The dimension is used to control the distribution range of attention scores. Corresponding to the cross attention module of this application, that is:

[0052] m1=CrossAtt(f1,f2)

[0053] m2=CrossAtt(f2,f1)

[0054] like Figure 2 As shown, in the technical solution of the present application, f1 and m1 are multiplied by corresponding spatial pixels to obtain G1. Similarly, f2 and m2 are multiplied by corresponding spatial pixels to obtain G2. After the above operations, G1 and G2 actually store the weighted multiplication results between the interaction information and the original feature map respectively, which strengthens the important feature information. Therefore, it is beneficial to improve the accuracy of the image matching learning network in the subsequent training process. For the present application, the result of learning is a transformation matrix containing several parameters. Only when the learning accuracy and precision of the matrix parameters are high can it be guaranteed that the image can be well aligned with the underlying visible light image after applying the transformation matrix. This is special compared to other classification tasks or learning tasks, that is, the training result requires a transformation matrix of several real number parameters, and its numerical changes have a significant impact on the accuracy of image matching alignment. Strengthening important information on the basis of m1 and m2 will help improve the accuracy. For the application scenario of the present application, its contribution to the accuracy of image matching is necessary.

[0055] As for the S3 feature learning step, the design of the image matching learning network is mainly considered. In addition, as shown in Figure 2 , it includes a pre-pooling layer and a post-network. The relevant outputs obtained from S1 and S2 first enter the pre-pooling layer for processing, and then the post-network is processed to obtain the final conversion matrix result. In order to learn and determine the conversion matrix required for image matching from the feature map, the model accuracy of the image matching learning network is required to be high. As shown in Figure 2 , the pre-pooling layer is composed of a pyramid structure of 3 levels of multi-scale (1*1, 2*2, 4*4) max-pooling layers. Through max-pooling, the size of the feature map can be effectively reduced, the amount of calculation and memory requirement can be reduced, the network can be more efficient when processing larger images, and it is helpful for the network to focus on important features in the image and ignore some less important information, thereby improving the efficiency of the network.

[0056] The post-network is composed of convolutional layers and max-pooling layers. The convolutional layers can further abstract and extract the input features. Among them, the convolutional layer with a kernel size of 1 pixel can perform linear transformation in the channel dimension without changing the spatial size of the feature map, which is helpful for fusing and filtering different feature information. On the other hand, the pooling layer reduces the spatial resolution of the feature map to compress the features, reduce the number of parameters and the amount of calculation, and extract more representative features, and finally output the required results. In the image matching learning network, in order to combine the input to achieve better matching effect, the loss function of the network model is improved, and an edge judgment influence factor is introduced to improve the matching effect. Assuming that the initial image feature block, the matched feature block and the unmatched feature block are a, p and n, the improved loss function L is defined as:

[0057] L = βP an - γP ap + G

[0058] The feature block is a small block area with specific features extracted from the input image, containing different image information such as color, texture, shape, etc. The positive sample is the feature block matched with the initial image feature block. In the two images to be matched, the feature blocks of the same or similar objects are positive samples. The negative sample is the feature block that does not match the initial image feature block. The negative sample usually comes from a different image or an unrelated area in the same image.

[0059] The feature block is obtained by dividing and extracting the input image, and the Euclidean distance between the positive and negative samples and the initial image feature block affects the calculation of the loss function, thereby affecting the output conversion matrix.

[0060] P ap = ||f(a)-f(p)||2and P an=||f(a)-f(n)||2 represents the Euclidean distance between the two image feature descriptions. f represents the corresponding deep network, whose structure is as follows Figure 2 As shown in the figure, Conv represents the convolution layer. β>0 and γ>0 are weights that control the rate of change of the loss function. In this embodiment, the measurement effect is better when β=2 and γ=0.3. G is the edge judgment influencing factor, and the specific formula is as follows:

[0061]

[0062] Wherein λ is an adjustment parameter used to control the weight of the edge influence factor. In this embodiment, when λ=10, the measurement effect is better.

[0063] This method uses feature scales of different specifications to extract image features, and then learns together through the correlation coefficients of different images to form a learning network for feature recognition, and finally achieves accurate registration of power equipment images. The above method realizes the registration of power equipment images, which has the advantages of high registration accuracy and wide applicability, and can be widely used in the field of power equipment safety monitoring. The present invention combines multi-specification feature scales to first learn the small-scale features in the image, and then realizes the overall learning of image features for large-scale features, introduces a cross-attention module between image blocks, calculates the information interaction between two points in different image blocks, and fuses the correlation coefficients between color images and thermal imaging images for similarity learning, and finally achieves accurate registration of color images and thermal imaging images. Compared with traditional methods, the present invention has the advantages of high registration accuracy, strong robustness, and wide applicability, and can be widely used in power equipment safety monitoring.

[0064] The above content only describes the preferred embodiments of the present invention and does not limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made to the technical solution of the present invention by ordinary technicians in this field should fall within the scope of protection determined by the claims of the present invention.

Claims

1. A power equipment image registration method based on deep learning, characterized in that: The steps include: S1, image feature extraction step, including the following feature extraction branches: The visible light image feature extraction branch takes the visible light image as input and obtains the feature map of the visible light image after processing by the visible light image feature extraction network; The thermal imaging image feature extraction branch takes the thermal imaging image as input and obtains the feature map of the thermal imaging image after processing by the thermal imaging image feature extraction network; The edge-annotated visible light image feature extraction branch takes as input a visible light image with key edge regions annotated. After processing by the additional image feature extraction network, the image feature map with additional edge annotations is obtained. S2. Applying a cross-attention mechanism: Applying a cross-attention module to obtain first interaction information between a feature map of the thermal image and a feature map of the visible light image, and second interaction information between a feature map of the visible light image and a feature map of the thermal image, respectively. Determine a correlation coefficient matrix between the thermal image and the visible light image based on the first interaction information and the second interaction information. S3, feature learning step: input the image feature map obtained in the image feature extraction step S1 and the correlation coefficient matrix obtained by applying the cross attention mechanism in S2 into the image matching learning network to obtain the conversion matrix result of converting the thermal imaging image into the visible light image coordinate system; In the step S2 where the cross attention mechanism is applied, the calculation formula for the interaction information is: Among them, Q and K represent image features, and d is QK T Dimensions, The first mutual information is defined as m1=CrossAtt(f1,f2); The second mutual information is defined as m2=CrossAtt(f2,f1); Where f1 represents the feature map of the visible light image, and f2 represents the feature map of the thermal imaging image; In the step S2 of applying the cross-attention mechanism, the correlation coefficient matrix G2 of the thermal imaging image is a feature weighted matrix calculated by multiplying the feature map f2 of the thermal imaging image and the corresponding pixels of the second interactive information m2; the correlation coefficient matrix G1 of the visible light image is a feature weighted matrix calculated by multiplying the feature map f1 of the visible light image and the corresponding pixels of the first interactive information m1.

2. The method for power equipment image registration based on deep learning according to claim 1, characterized in that: Assuming that the initial image feature block, the matched feature block and the unmatched feature block are a, p, and n, the loss function L of the image matching learning network is defined as: L=βP an -γP ap Among them, P ap =||f(a)-f(p)||2 and P an =||f(a)-f(n)||2 represents the Euclidean distance between the two image feature descriptions, β>0 and γ>0 are weights that control the rate of change of the loss function.

3. The method for power equipment image registration based on deep learning according to claim 1, characterized in that: Assuming that the initial image feature block, the matched feature block and the unmatched feature block are a, p, and n, the loss function L of the image matching learning network is defined as: L=βP an -γP ap +G Among them, P ap =||f(a)-f(p)||2 and P an =||f(a)-f(n)||2 represents the Euclidean distance between the two image feature descriptions; G is the edge judgment influencing factor, β>0 and γ>0 are the weights that control the rate of change of the loss function.

4. The method for power equipment image registration based on deep learning according to claim 3, characterized in that: The edge judgment influencing factor G is: Where λ is the adjustment parameter.

Images