Infrared-visible light image fusion method based on deformation field, storage medium and electronic equipment

Abstract

The invention relates to an infrared-visible light image fusion method based on a deformation field, a storage medium and electronic equipment. The infrared-visible light image fusion method comprises the following steps: acquiring an infrared image and a visible light image of a high-voltage switch cabinet; processing the visible light image by adopting a cross-modal perception style migration network in the multi-modal image registration model to generate a pseudo-infrared image; inputting the pseudo infrared image and the infrared image into a multistage refined registration network of the model to generate a distortion displacement vector deformation field, and performing registration reconstruction on the infrared image to obtain a registered infrared image; and respectively inputting the registered infrared image and visible light image into a feature extraction network, extracting features, fusing the features by adopting a multi-modal image fusion network, and obtaining and outputting a fused image. According to the invention, the deformation field is generated through the pseudo-infrared image to realize the registration and fusion of the infrared and visible light images, the method is constructed and applied to the infrared and visible light image fusion of the switch cabinet, the image registration and fusion are automatically realized, and the manual workload is reduced.

Description

Technical Field

[0001] This invention belongs to the field of high-voltage electrical equipment monitoring, specifically relating to an infrared-visible light image fusion method based on deformation field, a storage medium, and an electronic device. Background Technology

[0002] High-voltage switchgear is a core piece of equipment ensuring the safe and stable operation of power systems. Potential thermal faults within it (such as poor contact, overload, and insulation aging accompanied by overheating) are a major cause of equipment accidents. Infrared thermal imaging technology can visually display the temperature distribution of equipment and effectively detect abnormal heat points, but its low spatial resolution and lack of detailed equipment structure make it difficult for maintenance personnel to accurately locate heat points to specific physical components within the switchgear (such as specific bolts, contacts, or busbar connectors). While visible light imaging can provide high-resolution information about the equipment's structure, it cannot directly reflect temperature anomalies.

[0003] Therefore, precise registration and information fusion of infrared thermal images and visible light images to form a visualized image with "thermal-structural" overlay is of irreplaceable value for accurately locating heat sources, understanding the correspondence between thermal anomalies and equipment structure, and guiding efficient maintenance. However, currently, image registration and fusion in switchgear scenarios mainly face the following challenges and limitations: 1. Low level of automation: Current fusion solutions mainly rely on manual labor, which is inefficient and lacks automated means for feature point segmentation.

[0004] 2. Poor adaptability of the method: It requires determining the lens parameters of the infrared and visible light sensors to achieve fusion, but it cannot fuse images without parameters. Therefore, a reliable method is urgently needed to solve the current problem. Summary of the Invention

[0005] To overcome the aforementioned problems in the prior art, the present invention provides an infrared-visible light image fusion method, storage medium, and electronic device based on deformation field, which are used to solve the aforementioned problems in the prior art.

[0006] An infrared-visible image fusion method based on deformation field, the method comprising: S1. acquiring infrared and visible light images of a high-voltage switchgear; S2. Construct a multimodal image registration model, and use the cross-modal sensing style transfer network in the model to process the visible light image to generate a pseudo-infrared image; S3. Input the pseudo-infrared image and the infrared image into the multi-level thinning registration network of the model, compare the pseudo-infrared image and the infrared image to generate a distortion displacement vector deformation field, and use the deformation field to register and reconstruct the infrared image to obtain the registered infrared image. S4. Input the registered infrared image and visible light image into the feature extraction network respectively to extract features, and fuse the features using a multimodal image fusion network to obtain a fused image; S5. Output the fused image.

[0007] In addition to the aspects described above and any possible implementations, a further implementation is provided in which the acquisition is performed using an infrared camera and a visible light camera.

[0008] In addition to the aspects described above and any possible implementation, a further implementation is provided in which the cross-modal perceptual style transfer network includes an image generator and an image discriminator. The image generator is used to process the visible light image to obtain the pseudo-infrared image, and the image discriminator is used to determine the pseudo-infrared image and retain it as the final pseudo-infrared image if it belongs to a set target category.

[0009] In addition to the aspects described above and any possible implementations, a further implementation is provided in which the image generator includes an encoder, a converter, and a decoder; the visible light image is fed into an encoder consisting of three convolutional layers, the encoder extracts features from the visible light image, and outputs a feature vector. The feature vector is input to a converter consisting of nine ResNet residual modules, which converts the feature vector into the feature vector required for pseudo-infrared image processing. Finally, the required feature vectors are... The input to the decoder is restored to low-level features by two deconvolutional layers, and then the low-level features are converted into the pseudo-infrared image by a convolutional layer.

[0010] In addition to the aspects and any possible implementations described above, a further implementation is provided in which the image discriminator is a convolutional neural network, including four convolutional layers and a one-dimensional output convolutional layer. The pseudo-infrared image is input to the four convolutional layers for processing to extract image features. The one-dimensional output convolutional layer outputs the image features and makes a judgment to determine whether it belongs to the target category.

[0011] In addition to the aspects described above and any possible implementation, a further implementation is provided, wherein the multi-level refined registration network includes a feature extraction network, two deformation field prediction components C2F-DFE, and a resampling layer. The feature extraction network extracts features from the pseudo-infrared image and the infrared image respectively to obtain pseudo-infrared image features and infrared image features. The pseudo-infrared image features and infrared image features are respectively input to the two deformation field prediction components C2F-DFE to obtain predicted deformation fields. The predicted deformation fields are input to the resampling layer and registered with the infrared image to obtain a registered infrared image.

[0012] As described above and in any possible implementation, a further implementation is provided in which each of the deformation field prediction components C2F-DFE includes a coarse deformation field prediction module connected to it. The system includes a fine deformation field prediction module, a coarse deformation field prediction module that predicts a coarse deformation field from an input image, and a fine deformation field prediction module that further predicts the coarse deformation field to obtain the predicted deformation field.

[0013] In addition to the aspects and any possible implementations described above, a further implementation is provided in which the multimodal fusion network includes an encoder network, a fusion network, and a decoder network, wherein the encoder network extracts features from the registered infrared image and visible light image to obtain an infrared image feature map and a visible light image feature map. The fusion network uses the L1 norm and Softmax strategy to fuse the infrared image feature map and the visible light image feature map to obtain an intermediate image; the intermediate image is then input into the decoder network for reconstruction to obtain the fused image.

[0014] The present invention also provides a computer storage medium storing a computer program, the computer program being executed by a processor to implement the method described.

[0015] The present invention also provides an electronic device, the electronic device comprising: Memory, which stores executable instructions; A processor that executes the executable instructions in the memory to implement the method.

[0016] Beneficial effects of the present invention This invention employs an infrared-visible image fusion method based on a deformation field, comprising: acquiring infrared and visible light images of a high-voltage switchgear; constructing a multimodal image registration model, processing the visible light image using a cross-modal sensing style transfer network within the model to generate a pseudo-infrared image; inputting the pseudo-infrared image and the infrared image into a multi-level refinement registration network of the model, comparing the pseudo-infrared image and the infrared image to generate a distortion displacement vector deformation field, and using the deformation field to register and reconstruct the infrared image to obtain a registered infrared image; inputting the registered infrared image and the visible light image into a feature extraction network respectively to extract features, fusing the features using a multimodal image fusion network to obtain a fused image; and finally outputting the fused image. This invention utilizes a deformation field generated from a pseudo-infrared image to achieve infrared and visible light image registration and fusion. Applying this method to the fusion of infrared and visible light images of a switchgear can automatically achieve image registration and fusion, reducing manual workload. Furthermore, compared to similar algorithms, this method does not require manual setting of lens parameters, making it more versatile. Attached Figure Description

[0017] Figure 1 is a structural diagram of the multimodal image registration module of the present invention; Figure 2 This is a structural diagram of the multimodal image fusion network of the present invention. Detailed Implementation

[0018] To better understand the technical solution of this invention, the content of this invention includes, but is not limited to, the specific embodiments described below. Similar technologies and methods should be considered within the scope of protection of this invention. To make the technical problems to be solved, the technical solutions, and advantages of this invention clearer, a detailed description will be provided below in conjunction with the accompanying drawings and specific embodiments.

[0019] It should be understood that the embodiments described in this invention are merely some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without inventive effort are within the scope of protection of this invention.

[0020] The terminology used in the embodiments of this invention is for the purpose of describing particular embodiments only and is not intended to limit the invention. The singular forms “a,” “the,” and “the” as used in the embodiments of this invention and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.

[0021] This invention provides an infrared-visible image fusion method based on deformation field, the method comprising: S1. acquiring infrared and visible light images of a high-voltage switchgear; S2. Construct a multimodal image registration model, and use the cross-modal sensing style transfer network in the model to process the visible light image to generate a pseudo-infrared image; S3. Input the pseudo-infrared image and the infrared image into the multi-level thinning registration network of the model, compare the pseudo-infrared image and the infrared image to generate a distortion displacement vector deformation field, and use the deformation field to register and reconstruct the infrared image to obtain the registered infrared image. S4. Input the registered infrared image and visible light image into the feature extraction network respectively to extract features, and fuse the features using a multimodal image fusion network to obtain a fused image; S5. Output the fused image.

[0022] Furthermore, the data acquisition is performed using an infrared camera and a visible light camera.

[0023] Furthermore, the cross-modal perceptual style transfer network includes an image generator and an image discriminator. The image generator is used to process the visible light image to obtain the pseudo-infrared image, and the image discriminator is used to determine the pseudo-infrared image and retain it as the final pseudo-infrared image if it belongs to a set target category.

[0024] Furthermore, the image generator includes an encoder, a converter, and a decoder; the visible light image is fed into an encoder consisting of three convolutional layers, which extracts features from the visible light image and outputs a feature vector. The feature vector is input to a converter consisting of nine ResNet residual modules, which converts the feature vector into the feature vector required for pseudo-infrared image processing. Finally, the required feature vectors are... The input to the decoder is restored to low-level features by two deconvolutional layers, and then the low-level features are converted into the pseudo-infrared image by a convolutional layer.

[0025] Furthermore, the image discriminator is a convolutional neural network, including four convolutional layers and a one-dimensional output convolutional layer. The pseudo-infrared image is input to the four convolutional layers for processing to extract image features. The one-dimensional output convolutional layer outputs the image features and makes a judgment to determine whether it belongs to the target category.

[0026] Furthermore, the multi-level refinement registration network includes a feature extraction network, two deformation field prediction components C2F-DFE, and a resampling layer. The feature extraction network extracts features from the pseudo-infrared image and the infrared image respectively to obtain pseudo-infrared image features and infrared image features. The pseudo-infrared image features and infrared image features are then input into the two deformation field prediction components C2F-DFE to obtain the predicted deformation fields. The predicted deformation fields are then input into the resampling layer and registered with the infrared image to obtain the registered infrared image.

[0027] Furthermore, each of the deformation field prediction components C2F-DFE includes a coarse deformation field prediction module connected to it. The system includes a fine deformation field prediction module, a coarse deformation field prediction module that predicts a coarse deformation field from an input image, and a fine deformation field prediction module that further predicts the coarse deformation field to obtain the predicted deformation field.

[0028] Furthermore, the multimodal fusion network includes an encoder network, a fusion network, and a decoder network. The encoder network extracts features from the registered infrared image and visible light image to obtain infrared image feature maps and visible light image feature maps. The fusion network uses the L1 norm and Softmax strategy to fuse the infrared image feature map and the visible light image feature map to obtain an intermediate image; the intermediate image is then input into the decoder network for reconstruction to obtain the fused image.

[0029] Specifically, the processing flow for infrared and visible light image fusion in this invention is as follows: Step 1: Collect data from infrared and visible light cameras and input them into an image fusion network. The purpose is to fuse infrared and visible light images into a single image. Infrared lenses and visible light cameras are used to acquire the relevant data. There are no special requirements for the two types of cameras; they only need to be able to acquire images.

[0030] Step 2: Use the cross-modal sensing style transfer network in the multimodal image registration model to process the visible light image and generate a pseudo-infrared image. The purpose is to convert the visible light image into a pseudo-infrared image to unify the modal gap between the visible light and infrared images. Step 3: Input the pseudo-infrared image and the infrared image into the multi-level thinning registration network. Generate a distortion displacement vector field by comparing the pseudo-infrared image and the infrared image, and use the vector field to register and reconstruct the infrared image. The purpose is to obtain the distortion parameters of the infrared image relative to the visible light image. Because infrared cameras and visible light cameras are different, the distortion of their lenses is different. It is necessary to calculate the distortion before proceeding with subsequent operations. Otherwise, the image deformation will be different, and the operation cannot be performed.

[0031] Step 4: Finally, the registered infrared and visible light images are input into the feature extraction network. After feature extraction, the features are passed through a multimodal image fusion network to fuse the registered infrared and visible light images. This means that the two images are displayed in one, possessing the characteristics of two images. Specifically, the thermal radiation information of the registered infrared image and the structural details of the visible light image are fused to generate a "thermal-structural" superimposed image that combines temperature distribution and equipment texture, facilitating accurate fault location in the subsequent process.

[0032] Step 5: Output a fused image of visible light and infrared light.

[0033] like Figure 1 As shown in the diagram, this invention employs a multimodal image registration model. The purpose of this model is to register visible light and infrared images to generate a registered infrared image. The model consists of a cross-modal perceptive style transfer network (CPSMN) and a multi-scale refinement registration network (MRRN). The model first uses CPSMN to apply the model to visible images. Generating pseudo-infrared images and infrared images Together, they form an infrared image pair, which is then fed into the MRRN. The MRRN then... Predictive generation of infrared images The deformation field is used to generate the registered infrared image. .

[0034] The following describes the cross-modal perceptive style transfer network CPSMN and the multi-scale refinement registration network MRRN.

[0035] 1) Cross-modal perceptive style transfer network In infrared image registration, there are differences between images of different modalities. This invention uses a cross-modal sensing style transfer network (CPSMN) to convert visible light images into pseudo-infrared images to reduce the differences between cross-modal images.

[0036] The network structure of the Cross-Modal Aware Style Transfer Network (CPSMN) is shown below. Figure 1 CPSMN is a U-shaped network that generates pseudo-infrared images from visible light. and visible light generator that generates visible light from pseudo-infrared light Visible light image discriminator Pseudo-infrared image discriminator Since CPSMN focuses on generating pseudo-infrared images during application, the CPSMN network used in the network only retains the generator for generating pseudo-infrared images from visible light images during training. Therefore, this invention only uses the pseudo-infrared generator and pseudo-infrared discriminator for generating pseudo-infrared images from visible light.

[0037] like Figure 1 The generator consists of three parts: an encoder, a converter, and a decoder. In practice, it includes the following steps: First, the visible light image enters the encoder, which consists of three convolutional layers. The encoder reduces the image's dimensionality and increases its channel count through convolutional operations, thereby extracting features from the input image and finally outputting a feature vector. The first step includes image features of visible light; the second step involves processing the feature vectors obtained from the encoder. The input is fed into a transformer consisting of 9 layers of ResNet residual modules, which processes the feature vector. Different similar features are combined, and then the feature vectors needed to convert visible light images into pseudo-infrared images are determined based on the combined features. , For the one in front The features in the data are formed after passing through the ResNet module, that is, the feature vectors are processed. It underwent another round of processing; finally, it was... Input decoder, First, two deconvolutional layers are used to restore the low-level features, and then convolutional layers are used to reduce the dimensionality and convert them into pseudo-infrared images. The inverse convolution of vectors is transformed back into image form.

[0038] The discriminator is a convolutional neural network (CNN) whose function is to extract features from the image input to the discriminator. The input image is a fixed representation of a special image after processing through convolution and other operations. The discriminator determines whether the features belong to the target category, i.e., the category of the image being processed by the discriminator. For example, in a pseudo-infrared discriminator, it determines whether the image conforms to pseudo-infrared features. The discriminator first processes the input image through four convolutional layers to extract image features, and finally outputs the features through a one-dimensional output convolutional layer to determine whether it belongs to the target category.

[0039] 2) Multi-scale refined registration network A multi-scale refinement registration network vertically predicts the deformation field between the infrared image and the pseudo-infrared image, and generates the registered infrared image. For example... Figure 1As shown, MRRN consists of three parts. The first part is a feature extraction network shared by infrared and pseudo-infrared images, which extracts features from both images through image processing operations such as convolution. The second part consists of two deformation field prediction components (C2F-DFE). Each deformation field prediction component includes two modules: a coarse deformation field prediction module and a false deformation field prediction module. and a fine deformation field prediction module After the image enters the C2F-DFE, the coarse deformation field is first predicted according to formula (1), and then the fine deformation field is predicted according to formula (2): (1) (2) In the formula Refers to the feature extractor at the k-th scale. The operator represents element-wise summation. , These represent the outputs of the coarse deformation field and the fine deformation field, respectively. The result obtained after prediction is a matrix quantity of the outputs of the coarse deformation field and the fine deformation field, which records the offset of each point of the pseudo-infrared image relative to the infrared image. After these results are input back into the pseudo-infrared image, the pseudo-infrared image adjusts each pixel (according to the offset) to form a pseudo-infrared image that corresponds to the scale of the infrared image.

[0040] After predicting the deformation field, the network uses a resampling layer (the third part of the network) to calculate the final deformation field. and infrared images The registered infrared image is reconstructed using formula (3). .

[0041] (3) in Represents deformable fields, operators This indicates the spatial variations used for registration.

[0042] 3) Feature extraction network The feature extraction network is a multi-layer convolutional neural network. The image passes through an encoder network consisting of a convolutional batch normalization (CBS) activation layer and a dense block structure. The CBS module includes a convolutional conv layer, a batch normalization (BN) layer, and an activation layer. After the image is input into the convolutional layer, the convolutional layer performs a convolution operation on it. For example, if the input image is 32x32x3, it becomes 16x16x256 after the convolutional layer operation, and then enters the BN layer, which can be understood as normalizing the data. The activation function is calculated based on its function calculation formula to calculate the input matrix variables.

[0043] DenseBlock consists of three convolutional layers. Each layer can take the feature maps from all previous layers as its input, and the feature maps from this layer can then become the input for subsequent layers. This effectively enhances feature propagation, reduces the number of parameters, alleviates gradient vanishing, and preserves depth features. While convolutional layers reduce dimensionality by increasing the number of channels (e.g., from 32x32x3 to 16x16x256), DenseBlock only increases the number of channels without dimensionality reduction. The input is a matrix, and the output is also a matrix. Since the encoder network outputs to the image fusion network, which requires high image detail, pooling layers (which reduce image detail) are not used. When the encoder network extracts features, the image size remains unchanged, but the dimensionality increases. The purpose of the feature extraction network is to receive the registered infrared image. and visible light images The two images are converted into high-dimensional feature maps through several convolutions. Finally, the feature maps of the infrared image and the visible light image output by the encoder network are fed into the multimodal image fusion network for image fusion.

[0044] The purpose of this invention is to use a multimodal fusion network to fuse infrared and visible light images to generate a fused image. For example... Figure 2 As shown, the multimodal fusion network includes an encoder network, a fusion network, and a decoder network. The encoder of the multimodal fusion network used in this invention is the same as the encoder of the feature extraction network described earlier. The encoder network extracts features from the image, obtaining feature maps of the infrared and visible light images. and Then, these two feature maps are loaded into the fusion network part.

[0045] The fusion network part uses L1 norm and Softmax strategies to fuse images, such as... Figure 2 As shown, Infrared feature map and visible light characteristic map feature map of the set Represents the initial active graph and This represents the final active graph. For the final fusion result, the calculation formulas (4) and (5) are as follows: (4) It represents a set of feature maps, which are known quantities; (5) In the formula, The initial active map is N, the number of feature maps is N, (x,y) represents the corresponding position in the feature map at the current iteration number, r represents the block size, and in this invention, r=1 is chosen, which means that a pixel is obtained by averaging a 3x3 matrix centered on it, and a and b are both parameters.

[0046] The final active graph is obtained using formula (6). ,Will The output is sent to the decoding layer for image reconstruction.

[0047] (6) The decoding layer consists of four 3x3 convolutions, which affect the output of the fusion layer. Reconstruction is performed to obtain the reconstructed fused image. , For set In the equation, j, k, and n are parameter variables, with n taking values ​​from 1 to N.

[0048] Since the focus of the fusion network is on the encoder and decoder networks, which have better feature extraction and reconstruction capabilities, the fusion network is discarded during training, and only the encoder and decoder networks are trained.

[0049] As an embodiment of the present invention, the present invention also discloses a computer storage medium storing a computer program, the computer program being executed by a processor to implement the method described.

[0050] As an embodiment of the present invention, the present invention also discloses an electronic device, the electronic device comprising: Memory, which stores executable instructions; A processor that executes the executable instructions in the memory to implement the method.

[0051] The foregoing description illustrates and describes several preferred embodiments of the present invention. However, as previously stated, it should be understood that the present invention is not limited to the forms disclosed herein and should not be construed as excluding other embodiments. It can be used in various other combinations, modifications, and environments, and can be altered within the scope of the inventive concept described herein through the foregoing teachings or techniques or knowledge in related fields. Any modifications and variations made by those skilled in the art that do not depart from the spirit and scope of the present invention should be within the protection scope of the appended claims.

Claims

1. An infrared-visible image fusion method based on deformation field, characterized in that, The method includes: S1. Acquiring infrared and visible light images of the high-voltage switchgear; S2. Construct a multimodal image registration model, and use the cross-modal sensing style transfer network in the model to process the visible light image to generate a pseudo-infrared image; S3. Input the pseudo-infrared image and the infrared image into the multi-level thinning registration network of the model, compare the pseudo-infrared image and the infrared image to generate a distortion displacement vector deformation field, and use the deformation field to register and reconstruct the infrared image to obtain the registered infrared image. S4. Input the registered infrared image and visible light image into the feature extraction network respectively to extract features, and fuse the features using a multimodal image fusion network to obtain a fused image; S5. Output the fused image.

2. The method according to claim 1, characterized in that, The data acquisition was conducted using infrared and visible light cameras.

3. The method according to claim 1 or 2, characterized in that, The cross-modal sensing style transfer network includes an image generator and an image discriminator. The image generator is used to process the visible light image to obtain the pseudo-infrared image, and the image discriminator is used to determine the pseudo-infrared image and retain it as the final pseudo-infrared image if it belongs to a set target category.

4. The method according to claim 3, characterized in that, The image generator includes an encoder, a converter, and a decoder; the visible light image is fed into an encoder consisting of three convolutional layers, which extracts features from the visible light image and outputs a feature vector. The feature vector is input to a converter consisting of nine ResNet residual modules, which converts the feature vector into the feature vector required for pseudo-infrared image processing. Finally, the required feature vectors are... The input to the decoder is restored to low-level features by two deconvolutional layers, and then the low-level features are converted into the pseudo-infrared image by a convolutional layer.

5. The method according to claim 3, characterized in that, The image discriminator is a convolutional neural network, comprising four convolutional layers and a one-dimensional output convolutional layer. The pseudo-infrared image is input to the four convolutional layers for processing to extract image features. The one-dimensional output convolutional layer outputs the image features and makes a judgment to determine whether it belongs to the target category.

6. The method according to claim 1, characterized in that, The multi-level refinement registration network includes a feature extraction network, two deformation field prediction components (C2F-DFE), and a resampling layer. The feature extraction network extracts features from the pseudo-infrared image and the infrared image respectively to obtain pseudo-infrared image features and infrared image features. The pseudo-infrared image features and infrared image features are then input into the two deformation field prediction components (C2F-DFE) to obtain the predicted deformation fields. The predicted deformation fields are then input into the resampling layer and registered with the infrared image to obtain the registered infrared image.

7. The method according to claim 6, characterized in that, Each of the aforementioned deformation field prediction components C2F-DFE includes a coarse deformation field prediction module connected to it. The system includes a fine deformation field prediction module, a coarse deformation field prediction module that predicts a coarse deformation field from an input image, and a fine deformation field prediction module that further predicts the coarse deformation field to obtain the predicted deformation field.

8. The method according to claim 1, characterized in that, The multimodal fusion network includes an encoder network, a fusion network, and a decoder network. The encoder network extracts features from the registered infrared and visible light images to obtain infrared image feature maps and visible light image feature maps. The fusion network uses the L1 norm and Softmax strategy to fuse the infrared image feature map and the visible light image feature map to obtain an intermediate image; the intermediate image is then input into the decoder network for reconstruction to obtain the fused image.

9. A computer storage medium, characterized in that, The medium stores a computer program, which is executed by a processor to implement the method described in any one of claims 1-8.

10. An electronic device, characterized in that, The electronic device includes: Memory, which stores executable instructions; A processor that executes the executable instructions in the memory to implement the method of any one of claims 1-8.

Images