Power equipment image registration method based on deep learning

Abstract

The invention relates to a power equipment image registration method based on deep learning, and the method comprises an image feature extraction step which comprises a visible light image feature extraction branch; thermal imaging image feature extraction branches; extracting branches of visible light image features with marked edges; a step of applying a cross attention mechanism: obtaining first interaction information transmitted from the feature map of the thermal imaging image to the feature map of the visible light image and second interaction information transmitted from the feature map of the visible light image to the feature map of the thermal imaging image; in the feature learning step, the image feature map and the interaction information are input into an image matching learning network, and a matrix conversion result is obtained. According to the method, correlation coefficients of different images are learned, a learning network is formed for feature recognition, and finally accurate registration of the images of the power equipment is achieved. According to the method, the image registration of the power equipment is realized, and the method has the advantages of high registration precision, wide applicability and the like, and can be widely applied to the field of safety monitoring of the power equipment.

Description

Technical Field

[0001] The present invention relates to the technical field of image registration, and in particular to a method for image registration of electric power equipment based on deep learning. Background Art

[0002] With the introduction of the concept of smart grid and the widespread application of deep learning in the field of artificial intelligence, intelligent safety monitoring of power systems is becoming a research hotspot and development direction in the field of power scientific research and engineering applications. In recent years, the power industry has increasingly tended to unmanned monitoring of the safety of power equipment. The use of thermal imaging cameras to monitor power equipment in real time and then analyze and process the images to detect possible temperature anomalies has become an important part of the intelligentization of power systems.

[0003] In the scenario of unmanned monitoring of power grid equipment, it is feasible to use thermal imaging technology to monitor power grid equipment. However, the pixels of thermal imaging images are low. Although they can display the temperature change information in the scene, the recognition of scene objects is restricted by low resolution, and it is difficult to view the imaging details. Visible light imaging technology is already very mature and can achieve higher image resolution. During the maintenance process, color images can be used to zoom in and observe the equipment details in key areas. Therefore, in order to meet the needs of power grid abnormality monitoring and maintenance, it is necessary to provide a new monitoring scene display solution that integrates thermal imaging images and visible light images (black and white or color images), so that when the temperature is abnormal, the details of the abnormal local area can be quickly viewed and obtained, and the treatment plan for the abnormality can be determined. Summary of the invention

[0004] In the field of smart grid, thermal imaging technology based on temperature imaging of the power grid can detect temperature anomalies of power grid equipment, but its pixels are low, which is not conducive to finding and locating the cause of heating when abnormal, and rapid maintenance. To this end, the present invention provides a power equipment image registration method based on deep learning to achieve image registration fusion, thereby solving this technical problem.

[0005] The technical solution of the present invention provides a method for power equipment image registration based on deep learning, comprising the following steps: S1, image feature extraction step, including the following feature extraction branches: The visible light image feature extraction branch takes the visible light image as input and obtains the feature map of the visible light image after being processed by the visible light image feature extraction network; The thermal imaging image feature extraction branch takes the thermal imaging image as input and obtains the feature map of the thermal imaging image after being processed by the thermal imaging image feature extraction network; The edge-annotated visible light image feature extraction branch takes as input a visible light image with key edge regions annotated, and obtains an image feature map with edge annotations after being processed by an additional image feature extraction network; S2, step of applying a cross attention mechanism: applying a cross attention module to respectively obtain first interaction information from a feature map of the thermal imaging image to a feature map of the visible light image and second interaction information from a feature map of the visible light image to a feature map of the thermal imaging image, and combining the first interaction information and the second interaction information to determine a correlation coefficient matrix between the thermal imaging image and the visible light image; S3, feature learning step: input the image feature map obtained in the image feature extraction step S1 and the correlation coefficient matrix obtained in the cross-attention mechanism in S2 into the image matching learning network to obtain the conversion matrix result of the thermal imaging image into the visible light image coordinate system.

[0006] Preferably, in the step S2 of applying the cross attention mechanism, the calculation formula of the mutual information is:

[0007] in, , Represents image features. yes The dimension of The first interaction information is defined as ; The second interaction information is defined as ; Among them, f 1 Represents the feature map of the visible light image, f 2 Feature map representing a thermal image.

[0008] Preferably, in the step S2 of applying the cross-attention mechanism, the correlation coefficient matrix of the thermal imaging image is a feature weighting matrix calculated by multiplying a feature map of the thermal imaging image with corresponding pixels of the first interaction information; the correlation coefficient matrix of the visible light image is a feature weighting matrix calculated by multiplying a feature map of the visible light image with corresponding pixels of the second interaction information.

[0009] Preferably, assuming that the initial image feature block, the matched feature block and the unmatched feature block are a, p, n, the loss function L of the image matching learning network is defined as:

[0010] in and Represents the Euclidean distance between two image feature descriptions.

[0011] Preferably, the loss function L of the image matching learning network is defined as:

[0012] in and Represents the Euclidean distance between two image feature descriptions; It is the marginal judgment impact factor.

[0013] Preferably, the edge judgment influencing factor G is:

[0014] in To adjust the parameters.

[0015] This method uses feature scales of different specifications to extract image features, and then learns the correlation coefficients of different images together to form a learning network for feature recognition, and finally achieves accurate registration of power equipment images. The above method realizes the registration of power equipment images, which has the advantages of high registration accuracy and wide applicability, and can be widely used in the field of power equipment safety monitoring.

[0016] The present invention relates to the field of image processing technology, and in particular to a method for image registration of power equipment based on deep learning. A cross-attention module between image blocks is introduced, the information interaction between two points in different image blocks is calculated, the correlation coefficient between the color image and the thermal imaging image is fused for similarity learning, and finally the accurate registration of the color image and the thermal imaging image is achieved. Compared with the traditional method, the present invention has the advantages of high registration accuracy, strong robustness, wide applicability, etc., and can be widely used in the safety monitoring of power equipment. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 A schematic diagram of a flow chart of a method for image registration of power equipment based on deep learning according to the present invention; Figure 2 A schematic diagram of the principle of optimizing the application of the cross-attention mechanism of the present invention; Figure 3 It is the alignment result when the power equipment image registration method based on deep learning of the present invention adopts edge annotation information. DETAILED DESCRIPTION

[0018] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. In this specification, the size ratios in the drawings do not represent the actual size ratios, but are only used to reflect the relative position relationship and connection relationship between the components. Components with the same name or the same number represent similar or identical structures and are only for illustrative purposes.

[0019] To solve the problem that thermal imaging images can only determine the temperature distribution of equipment, but the resolution is insufficient and it is difficult to observe the details of the equipment, the method of fusing thermal imaging images and visible light images can be used in the design of smart grids. In actual installation scenarios, thermal imaging cameras and visible light cameras are generally not necessarily installed in the same place, so the image perspectives of the same equipment obtained by the two systems are often different. In addition, the complete sets of equipment for power equipment monitoring all adopt mature commercial solutions, among which the imaging resolution of color cameras has obvious advantages over the imaging resolution of thermal imaging images. In other words, the number of pixels of the two is very different. In view of the differences in shooting perspective and pixel size, it is difficult to directly fuse thermal imaging images with visible light images. To this end, the present invention provides a method for image registration of power equipment based on deep learning.

[0020] The image registration method of the present invention aims to determine the conversion matrix of the thermal imaging image to the visible light image through deep learning based on the input thermal imaging image and the visible light image. After the present invention obtains the conversion matrix, the thermal imaging image is rotated, scaled, translated, and the like by applying the conversion matrix to the thermal imaging image, thereby aligning the thermal imaging image to the visible light image, and finally merging it into the visible light image. The deep learning model used in the following image registration method is pre-trained using paired image data of pre-aligned thermal imaging images and visible light images (i.e., the conversion matrix is ​​determined). The model training process based on a specific model according to the required samples belongs to the prior art and will not be elaborated in this application.

[0021] In the application process, Figure 1 , 2 As shown, the deep learning-based power equipment image registration method of the present application generally includes the following steps.

[0022] S1, image feature extraction step. This step obtains the feature map corresponding to the input image by processing the input image through the feature extraction network. Specifically, it includes several branches that can be processed in parallel. These branches are: Visible light image feature extraction branch. The input is a visible light image. After being processed by the visible light image feature extraction network, the feature map f of the visible light image is obtained. 1 .

[0023] Thermal imaging image feature extraction branch. The input is a thermal imaging image. After being processed by the thermal imaging image feature extraction network, the feature map f of the thermal imaging image is obtained. 2 .

[0024] The visible light image feature extraction branch with edge annotation. The input is a visible light image with key edge areas annotated. After being processed by the additional image feature extraction network, an image feature map E with additional edge annotations is obtained.

[0025] S2. Apply the cross-attention mechanism step.

[0026] In order to improve the accuracy of the model’s learning for this task, the feature map f of the visible light image is 1 The characteristic map f of the thermal imaging image 2 The cross attention mechanism is applied between them to obtain the correlation coefficient between the two. That is, the feature maps f of the thermal imaging image are obtained by the cross attention module. 2 The feature map f transferred to the visible light image 1 The first interaction information m 1 And the feature map f of the visible light image 1 The feature map f transferred to the thermal imaging image 2 The second interaction information m 2 .

[0027] S3, feature learning step. The image feature map obtained in the image feature extraction step S1 and the correlation coefficient obtained in the cross-attention mechanism in S2 are input into the image matching learning network, which includes several convolutional layers and several pre-pooling layers. After processing, the result of deep learning is obtained, that is, the conversion matrix result of the thermal imaging image into the visible light image coordinate system.

[0028] For the S1 image feature extraction step, the sources of thermal imaging and visible light images can be taken from thermal imaging cameras and visible light cameras arranged on site. Since the camera outputs a video stream of more than 20fps, the thermal imaging image and visible light image at the same time are obtained by extracting frames from the video stream during actual processing. In this scheme, pairs of thermal imaging images and visible light images are obtained by randomly extracting 5 frames per second. In order to facilitate the matching calculation between image features in the feature extraction and feature learning process, it is necessary to process the input pairs of thermal imaging images and visible light images. Generally, it is necessary to adjust the image size and resolution to obtain a uniform input image size. On this basis, the contour edges in the visible light image are analyzed and extracted through the pre-trained edge feature extraction module, focusing on retaining the edge information of the power equipment, and deleting the contour edges of the identification background pattern. Its main purpose is to increase the weight of feature learning of the power equipment of interest in the feature extraction and feature learning process by additionally marking the edge information of the power equipment, so that the model has a higher accuracy rate for image registration in the power equipment scene, and to a certain extent accelerate the processing process of the model and improve the processing efficiency.

[0029] Since the input images are uniformly processed and have the same size and resolution, in order to ensure that the dimensions of the output image feature maps match, the feature extraction networks used preferably have the same network architecture. Of course, the weight parameters of each convolution kernel in the specific convolution layer are determined by the training process and are not necessarily the same. The input image size and the output feature map size are 640*480 pixels. After a series of convolution operations, the small-scale feature description network generates a specific number of feature channels. When the output of the small-scale network is passed to the large-scale network, the channel number information is also passed, and a larger receptive field (3*3 pixels -> 7*7 pixels) is used. Neurons will be able to capture a larger range of areas on the input feature map, which helps to capture a wider range of contextual information and is more conducive to extracting global structural information in the image.

[0030] The feature extraction network is set up with two description networks with different feature scales (receptive fields of 3*3 and 7*7 pixels respectively). The small-scale feature (3*3 pixels) description network is trained first, and then the large-scale feature (7*7 pixels) description network is trained. The output feature map of the small-scale feature description network is used as the input of the large-scale feature description network to obtain the feature maps after processing different input images. Feature scale refers to the scale used to describe information in different spatial ranges in an image. Generally, feature scale can be achieved by applying filters of different sizes in a convolutional neural network. Filters of different sizes can capture features of different scales in an image, such as edges, textures, and the overall structure of an object. Feature scales of different specifications affect the extraction and expression capabilities of features. Smaller feature scales are usually more suitable for capturing local detail information in an image, such as textures and edges, while larger feature scales are more suitable for capturing global structural information, such as the overall shape and layout of an object. Therefore, using feature scales of different specifications can provide a more comprehensive and diverse image representation, thereby improving the richness and expression capabilities of features.

[0031] like Figure 3 As shown in the figure, the thermal imaging grayscale image can mainly identify the area where the equipment temperature is abnormal, but since the background temperature in the thermal imaging image is not much different, the edge of the equipment cannot be clearly displayed on the image. In view of actual needs, we need to align the main body of the power equipment in the thermal imaging grayscale image with the main body of the power equipment in the color image, so as to align the equipment outline of the color image to the thermal imaging image and finally obtain the accurate equipment edge outline in the thermal imaging grayscale image (thermodynamic image). Thermal imaging grayscale images have the disadvantages of blurred images, more noise points and unclear equipment edges, such as Figure 3As shown in the figure, the experiment found that the conventional alignment results without edge annotation have large deviations, and the edge contours are obviously deviated from the device edges in the thermodynamic image, so it is difficult to be effectively applied in the production environment. When the deep learning method with edge annotation is used, the edge alignment effect is significantly better than that without edge annotation information.

[0032] In the step S2 of applying the cross attention mechanism, in order to improve the recognition of the correlation between the thermal imaging image and the visible light image, and thus improve the recognition efficiency and accuracy of the image subject, the feature map f of the thermal imaging image obtained in the step S1 of image feature extraction is 2 And the feature map f of the visible light image 1 Apply the criss-cross attention mechanism.

[0033] f 1 and f 2 Enter the criss-cross attention module, which is used to calculate f 1 and f 2 The reorganization mapping of the eigenvectors between any two points is recorded as and , yes Deliver to Information, yes Deliver to The specific calculation formula is as follows:

[0034] in, , Represents image features. yes The dimension is used to control the distribution range of the attention score. Corresponding to the cross attention module of this application, that is:

[0035]

[0036] like Figure 2 As shown, in the technical solution of this application, f 1 and m 1 G is obtained by corresponding multiplication on spatial pixels. 1 Similarly, f 1 and m 2 G is obtained by corresponding multiplication on spatial pixels. 2 After the above operation, G 1 With G 2In fact, the weighted multiplication results between the interaction information and the original feature map are stored separately, which strengthens the important feature information. Therefore, it is beneficial to improve the accuracy of the image matching learning network in the subsequent training process. For this application, the result of learning is a transformation matrix containing several parameters. Only when the learning accuracy and precision of the matrix parameters are high can it be guaranteed that the image can be well aligned with the underlying visible light image after applying the transformation matrix. This is special compared to other classification tasks or learning tasks, that is, the training result requires a transformation matrix of several real number parameters, and its numerical changes have a significant impact on the accuracy of image matching alignment. By 1 With m 2 Strengthening important information on this basis will be beneficial to improving accuracy. For the application scenarios of this application, its contribution to the accuracy of image matching is necessary.

[0037] As for the S3 feature learning step, it mainly involves the design of the image matching learning network. Figure 2 As shown in , it includes a pre-pooling layer and a post-network. The relevant outputs obtained from S1 and S2 are first processed by the pre-pooling layer, and then processed by the post-network to obtain the final conversion matrix result. In order to learn and determine the conversion matrix required for image matching from feature maps, the model accuracy of the image matching learning network is required to be high. Figure 2 As shown in the figure, the pre-pooling layer consists of a pyramid structure of 3-level multi-scale (1*1, 2*2, 4*4) maximum pooling layers. Through maximum pooling, the size of the feature map can be effectively reduced, the amount of calculation and memory requirements can be reduced, making the network more efficient when processing larger images, and helping the network focus on important features in the image and ignore some less important information, thereby improving the performance of the network.

[0038] The post-network consists of convolutional layers and maximum pooling layers. The convolutional layer can further abstract and extract the input features. The convolutional layer with a convolution kernel size of 1 pixel can perform linear transformations in the channel dimension without changing the spatial size of the feature map, which helps to fuse and filter different feature information. On the other hand, the pooling layer compresses the features by reducing the spatial resolution of the feature map, reduces the number of parameters and the amount of calculation, and extracts more representative features, and finally outputs the required results. In the image matching learning network, in order to achieve better matching effects combined with the input, the loss function of the network model is improved, and the edge judgment influencing factor is introduced to improve the matching effect. Assuming that the initial image feature block, the matched feature block, and the unmatched feature block are a, p, and n, the improved loss function L is defined as:

[0039] Feature blocks are small areas with specific features extracted from the input image, containing different image information, such as color, texture, shape, etc. Positive samples are feature blocks that match the feature blocks of the initial image. Feature blocks of the same or similar objects in the two images to be matched are positive samples. Negative samples are feature blocks that do not match the feature blocks of the initial image. Negative samples usually come from images different from the initial image or irrelevant areas in the same image.

[0040] The feature blocks are obtained by dividing and extracting the input image. The Euclidean distance between the positive and negative samples and the initial image feature blocks affects the calculation of the loss function, thereby affecting the output transformation matrix.

[0041] and Represents the Euclidean distance between two image feature descriptions. Represents the corresponding deep network, whose structure is as follows Figure 2 As shown, Conv represents the convolutional layer. and is the weight that controls the rate of change of the loss function. In this embodiment and The time measurement effect is better. is the edge judgment influencing factor, and the specific formula is as follows:

[0042] in is an adjustment parameter used to control the weight of the edge influence factor. The time measurement effect is better.

[0043] This method uses feature scales of different specifications to extract image features, and then learns together through the correlation coefficients of different images to form a learning network for feature recognition, and finally realizes the accurate registration of power equipment images. The registration of power equipment images is realized by the above method, which has the advantages of high registration accuracy and wide applicability, and can be widely used in the field of power equipment safety monitoring. The present invention combines multi-specification feature scales to first learn the small-scale features in the image, and then realizes the overall learning of image features for large-scale features, introduces a cross-attention module across image blocks, calculates the information interaction between two points in different image blocks, and fuses the correlation coefficients between color images and thermal imaging images for similarity learning, and finally realizes the accurate registration of color images and thermal imaging images. Compared with traditional methods, the present invention has the advantages of high registration accuracy, strong robustness, and wide applicability, and can be widely used in power equipment safety monitoring.

[0044] The above content only describes the preferred implementation mode of the present invention, and does not limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made to the technical solution of the present invention by ordinary technicians in this field should fall within the protection scope determined by the claims of the present invention.

Claims

1. A method for power equipment image registration based on deep learning, characterized in that: The steps include: S1, image feature extraction step, including the following feature extraction branches: The visible light image feature extraction branch takes the visible light image as input and obtains the feature map of the visible light image after being processed by the visible light image feature extraction network; The thermal imaging image feature extraction branch takes the thermal imaging image as input and obtains the feature map of the thermal imaging image after being processed by the thermal imaging image feature extraction network; The edge-annotated visible light image feature extraction branch takes as input a visible light image with key edge regions annotated, and obtains an image feature map with edge annotations after being processed by an additional image feature extraction network; S2, step of applying a cross attention mechanism: applying a cross attention module to respectively obtain first interaction information from a feature map of the thermal imaging image to a feature map of the visible light image and second interaction information from a feature map of the visible light image to a feature map of the thermal imaging image, and combining the first interaction information and the second interaction information to determine a correlation coefficient matrix between the thermal imaging image and the visible light image; S3, feature learning step: input the image feature map obtained in the image feature extraction step S1 and the correlation coefficient matrix obtained in the cross-attention mechanism in S2 into the image matching learning network to obtain the conversion matrix result of the thermal imaging image into the visible light image coordinate system.

2. The method for power equipment image registration based on deep learning according to claim 1, characterized in that: In the step of applying the cross attention mechanism in S2, the calculation formula of the mutual information is: in, , Represents image features. yes The dimension of The first interaction information is defined as ; The second interaction information is defined as ; Among them, f1 represents the feature map of the visible light image, and f2 represents the feature map of the thermal imaging image.

3. The method for power equipment image registration based on deep learning according to claim 2, characterized in that: In the step S2 of applying the cross-attention mechanism, the correlation coefficient matrix of the thermal imaging image is a feature weighted matrix calculated by multiplying the feature map of the thermal imaging image with the corresponding pixels of the first interactive information; the correlation coefficient matrix of the visible light image is a feature weighted matrix calculated by multiplying the feature map of the visible light image with the corresponding pixels of the second interactive information.

4. The method for power equipment image registration based on deep learning according to claim 1, characterized in that: Assuming that the initial image feature block, the matched feature block and the unmatched feature block are a, p, and n, the loss function L of the image matching learning network is defined as: in and represents the Euclidean distance between two image feature descriptions, and is the weight that controls the rate of change of the loss function.

5. The method for power equipment image registration based on deep learning according to claim 1, characterized in that: The loss function L of the image matching learning network is defined as: in and Represents the Euclidean distance between two image feature descriptions; is the marginal judgment impact factor, and is the weight that controls the rate of change of the loss function.

6. The method for power equipment image registration based on deep learning according to claim 1, characterized in that: The edge judgment influencing factor G is: in To adjust the parameters.

Images