Infrared image and depth image fusion method, system, device and storage medium

Abstract

The present invention belongs to the technical field of power equipment status monitoring, and specifically relates to a method and system for fusing infrared images and depth images based on an attention mechanism. In view of the low accuracy of existing power equipment status recognition methods, the present invention adopts the following technical solution: a method for fusing infrared images and depth images based on an attention mechanism, which is applied to a pre-constructed generative adversarial network, wherein the generative adversarial network includes a generator, an attention feature extraction module and a discriminator, including: obtaining an infrared image and a depth image, encoding them separately; extracting attention features from the encoded depth image to obtain a fused image; detecting the fused image to obtain infrared image details and depth image details. The beneficial effects of the present invention are: the use of depth images can adapt to image focal length differences, suppress interference factors such as sunlight and environment, take into account the requirements of enhancing key areas and retaining equipment detail information, and improve the accuracy of equipment status monitoring.

Description

Technical Field

[0001] The present invention belongs to the technical field of power equipment status monitoring, and specifically relates to an infrared image and depth image fusion method, system, device and storage medium based on an attention mechanism. Background Art

[0002] The reliability of power equipment is the basis for the safe operation of the power grid, and the perception of its operating status is an important support for the operation and maintenance of the power grid and operation control. In recent years, the state perception technology of power equipment has developed rapidly in the direction of digitalization and intelligence, and new technologies have emerged continuously, effectively improving the intelligence level of equipment. The state perception of power equipment is mostly achieved by acquiring infrared images and visible light images.

[0003] Infrared images have strong target recognition capabilities and are more sensitive to heat source areas, which is conducive to discovering targets such as equipment heat points, and can achieve all-weather monitoring day and night. Visible light images can provide rich background information for the target area, so that the target distance, surface texture, environmental information, etc. can be better understood. Using artificial intelligence, deep learning and other technologies to analyze infrared and visible light images, the status of power equipment can be judged more accurately.

[0004] The Chinese invention patent application with publication number CN 111709902 A discloses a method for fusion of infrared and visible light images based on a self-attention mechanism. The method first builds a deep self-encoding network structure based on the self-attention mechanism, inputs the feature maps of different branches of the encoding layer into the self-attention mechanism to obtain the attention map, and adopts the mean fusion strategy to obtain the output feature map. Two types of loss functions, content loss and detail loss, are designed to highlight the infrared target information, sharpen the edges, and better utilize the texture details in the source image. Finally, the neural network is trained, and the self-attention mechanism is visualized to adjust the network structure and loss function. Although this method allocates attention in the best way by learning the attention map, obtains the key information of the image, improves the visual effect, and improves the quality of the fused image, it lacks the inspection of the fused image and cannot meet the high accuracy requirements of power equipment monitoring.

[0005] The Chinese invention patent application with publication number CN114187214A discloses an infrared and visible light image fusion system and method, which respectively extracts multi-scale features of infrared images and visible light images through a multi-scale feature extraction unit, then performs feature screening on the multi-scale features through a dual attention mechanism, and then completes image fusion by an image reconstruction unit, thereby obtaining an infrared and visible light fusion image with clear details. However, the above method lacks consideration of long-distance infrared, laser, visible light and other imaging methods. Due to the characteristics of the working environment of power equipment, especially in the long-distance non-contact measurement of overhead transmission lines, there are interferences such as image focal length, sunlight, wind deviation, etc. In addition to visible light images, it is also necessary to introduce analysis of depth images.

[0006] In summary, in order to improve the accuracy of power equipment status monitoring, especially for scenarios such as long-distance non-contact temperature measurement of transmission lines, it is necessary to realize the fusion of depth images based on the existing infrared and visible light image fusion; in order to improve the quality of the fused image, it is necessary to apply artificial intelligence algorithms and introduce attention mechanisms to better preserve the source image information and solve the problem of the integrity of key areas and details in the perceived image; in order to improve the accuracy of the fused image, it is necessary to use generative adversarial network technology and adopt a dual discriminator structure, taking into account the differences between infrared images and depth images, and obtaining a fused image through the game confrontation between the generator and the discriminator, which can adapt to the difference in imaging focal length, suppress interference factors such as sunlight and environment, and promote the digital and intelligent development of power equipment status monitoring technology. Summary of the invention

[0007] The present invention aims at the disadvantage that the existing power equipment status recognition method adopts the fusion of infrared image and visible light image, which is applied in the long-distance non-contact temperature measurement scene such as the power transmission line of the power equipment, and has low accuracy. The present invention provides an infrared image and depth image fusion method based on the attention mechanism, which fuses the infrared image and the depth image to improve the accuracy of equipment status monitoring. The present invention also provides an infrared image and depth image fusion system based on the attention mechanism, an image fusion device and a computer-readable storage medium.

[0008] To achieve the above object, the present invention adopts the following technical solution: a method for fusion of infrared image and depth image based on attention mechanism is applied to a pre-constructed generative adversarial network, the generative adversarial network includes a generator, an attention feature extraction module and a discriminator, and the method includes:

[0009] Step S101, obtaining an infrared image and a depth image of the electric power equipment, and encoding the infrared image and the depth image of the electric power equipment respectively based on the generator to obtain encoded infrared image and depth image of the electric power equipment;

[0010] Step S102: Based on the attention feature extraction module, extract attention features from the encoded infrared image and depth image of the power equipment to obtain power infrared features and depth image features, and fuse the power infrared features and the depth image features to obtain a fused image;

[0011] Step S103: using the discriminator to detect the fused image to obtain infrared image details and depth image details of the fused image.

[0012] The infrared image and depth image fusion method based on the attention mechanism of the present invention takes into account that the infrared image and depth image of power equipment have different modalities, introduces the attention mechanism in the image fusion process, extracts the corresponding features and realizes image fusion for applications such as non-contact temperature measurement of power equipment. Compared with visible light images, the use of depth images can adapt to the difference in image focal length, suppress interference factors such as sunlight and environment, and take into account the requirements of enhancing key areas and retaining equipment detail information, thereby improving the accuracy of equipment status monitoring.

[0013] Furthermore, in step S101, the generator includes two data streams, each data stream includes three convolution modules, and each convolution module includes a convolution layer and a ReLU activation function.

[0014] Furthermore, in step S102, the input feature is defined as Feature i , after the global average pooling operation, the key value Key of each feature is obtained i , define the image details to be fused as Layer, and associate the image details Layer with the key value Key i The similarity calculation is performed to obtain the weight coefficient, and then the weight is multiplied by the original feature, as follows:

[0015]

[0016] Based on the attention mechanism, the multi-scale perception ability of typical feature areas in the image is improved, and then the features are combined with a unified decoding process to generate a fused image.

[0017] Further, in step S103, the discriminator is a multi-layer network including convolution, there are two discriminators, the two discriminators have the same network structure, the discriminator includes 6 convolution modules, each module includes a convolution layer and a LeakyReLU activation function. Further, in step S103, the infrared image details include infrared intensity.

[0018] Further, in step S103, the depth image details include at least one of distance, contour, and texture.

[0019] Furthermore, the power equipment includes substation equipment and overhead lines; an infrared thermal imager or an optical zoom infrared imaging device is used to obtain infrared images, and a three-dimensional laser scanner is used to obtain depth images.

[0020] An infrared image and depth image fusion system based on an attention mechanism, the system comprising:

[0021] The encoding module is used to encode the infrared image and the depth image of the electric power equipment respectively to obtain the encoded infrared image and the depth image of the electric power equipment;

[0022] An attention feature extraction module is used to extract attention features from the encoded infrared image and depth image of the power equipment to obtain power infrared features and depth image features, and fuse the power infrared features and the depth image features to obtain a fused image;

[0023] The discrimination module is used to detect the fused image to obtain infrared image details and depth image details of the fused image.

[0024] Furthermore, the encoding module includes two data streams, each data stream includes three convolution modules, and each convolution module includes a convolution layer and a ReLU activation function.

[0025] Furthermore, the attention feature extraction module defines the input feature as Feature i , after the global average pooling operation, the key value Key of each feature is obtained i , define the image details to be fused as Layer, and associate the image details Layer with the key value Key i The similarity calculation is performed to obtain the weight coefficient, and then the weight is multiplied by the original feature, as follows:

[0026]

[0027] Based on the attention mechanism, the multi-scale perception ability of typical feature areas in the image is improved, and then the features are combined with a unified decoding process to generate a fused image.

[0028] Furthermore, the discriminant module includes a dual discriminator, the two discriminators have the same network structure, the discriminator is a multi-layer network including convolution, the discriminator includes 6 convolution modules, each module includes a convolution layer and a LeakyReLU activation function, which respectively check the infrared intensity and depth image details in the fused image, and the depth image details include at least one of distance, contour, and texture.

[0029] The beneficial effect of the infrared image and depth image fusion method based on the attention mechanism of the present invention is: taking into account that the infrared image and depth image of power equipment have different modalities, in the image fusion process, an attention mechanism is introduced on the basis of the generator and the discriminator, and for applications such as non-contact temperature measurement of power equipment, the corresponding features are extracted and image fusion is realized. Compared with visible light images, the use of depth images can adapt to image focal length differences, suppress interference factors such as sunlight and environment, and take into account the requirements of enhancing key areas and retaining equipment detail information, thereby improving the accuracy of equipment status monitoring. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 It is a flowchart of an infrared image and depth image fusion method based on an attention mechanism according to an embodiment of the present invention.

[0031] Figure 2 It is a schematic diagram of non-contact infrared temperature measurement of power transmission lines based on an infrared image and depth image fusion method of an attention mechanism according to an embodiment of the present invention.

[0032] Figure 3 It is a structural block diagram of an infrared image and depth image fusion system based on an attention mechanism according to an embodiment of the present invention.

[0033] Figure 4 It is a structural block diagram of the attention feature extraction module of the infrared image and depth image fusion system based on the attention mechanism in an embodiment of the present invention. DETAILED DESCRIPTION

[0034] The following is an explanation and description of the technical solutions of the embodiments of the present invention in conjunction with the accompanying drawings of the embodiments of the present invention, but the following embodiments are only preferred embodiments of the present invention, not all. Based on the embodiments in the implementation mode, other embodiments obtained by those skilled in the art without creative work are all within the protection scope of the present invention.

[0035] See also Figures 1 to 4 , a method for fusing infrared images and depth images based on an attention mechanism of the present invention, the method for fusing infrared images and depth images based on an attention mechanism comprising:

[0036] Step S101, obtaining an infrared image and a depth image of the electric power equipment, and encoding the infrared image and the depth image of the electric power equipment respectively based on the generator to obtain encoded infrared image and depth image of the electric power equipment;

[0037] Step S102: Based on the attention feature extraction module, extract attention features from the encoded infrared image and depth image of the power equipment to obtain power infrared features and depth image features, and fuse the power infrared features and the depth image features to obtain a fused image;

[0038] Step S103: using the discriminator to detect the fused image to obtain infrared image details and depth image details of the fused image.

[0039] The infrared image and depth image fusion method based on the attention mechanism of the present invention takes into account that the infrared image and depth image of power equipment have different modalities. In the image fusion process, an attention mechanism is introduced on the basis of the generator and the discriminator. For applications such as non-contact temperature measurement of power equipment, the corresponding features are extracted and image fusion is realized. Compared with visible light images, the use of depth images can adapt to the difference in image focal length, suppress interference factors such as sunlight and environment, and take into account the requirements of enhancing key areas and retaining equipment detail information, thereby improving the accuracy of equipment status monitoring.

[0040] System Example

[0041] See also Figures 1 to 4 , an infrared image and depth image fusion system based on attention mechanism of the present invention, the system comprises:

[0042] The encoding module is used to encode the infrared image and the depth image of the electric power equipment respectively to obtain the encoded infrared image and the depth image of the electric power equipment;

[0043] An attention feature extraction module is used to extract attention features from the encoded infrared image and depth image of the power equipment to obtain power infrared features and depth image features, and fuse the power infrared features and the depth image features to obtain a fused image;

[0044] The discrimination module is used to detect the fused image to obtain infrared image details and depth image details of the fused image.

[0045] In this embodiment, the encoding module includes two data streams, each data stream includes three convolution modules, and each convolution module includes a convolution layer and a ReLU activation function.

[0046] In this embodiment, the attention feature extraction module defines the input feature as Feature i , after the global average pooling operation, the key value Key of each feature is obtained i , define the image details to be fused as Layer, and associate the image details Layer with the key value Key i The similarity calculation is performed to obtain the weight coefficient, and then the weight is multiplied by the original feature, as follows:

[0047]

[0048] Based on the attention mechanism, the multi-scale perception ability of typical feature areas in the image is improved, and then the features are combined with a unified decoding process to generate a fused image.

[0049] In this embodiment, the discriminant module includes a dual discriminator, the two discriminators have the same network structure, the discriminator is a multi-layer network including convolution, the discriminator includes 6 convolution modules, each module includes a convolution layer and a LeakyReLU activation function, which respectively check the infrared intensity and depth image details in the fused image, and the depth image details include distance, contour, and texture.

[0050] Method Embodiment

[0051] See also Figures 1 to 4 , an infrared image and a depth image fusion method based on an attention mechanism according to an embodiment of the present invention, the infrared image and the depth image fusion method based on the attention mechanism comprising:

[0052] Step S101, obtaining an infrared image and a depth image of the electric power equipment, and encoding the infrared image and the depth image of the electric power equipment respectively based on the generator to obtain encoded infrared image and depth image of the electric power equipment;

[0053] Step S102: Based on the attention feature extraction module, extract attention features from the encoded infrared image and depth image of the power equipment to obtain power infrared features and depth image features, and fuse the power infrared features and the depth image features to obtain a fused image;

[0054] Step S103: using the discriminator to detect the fused image to obtain infrared image details and depth image details of the fused image.

[0055] In this embodiment, in step S101, the generator is based on two data streams, each data stream contains three convolution modules, and each convolution module includes a convolution layer and a ReLU activation function.

[0056] In this embodiment, in step S102, the input feature is defined as Feature i , after the global average pooling operation, the key value Key of each feature is obtained i , define the image details to be fused as Layer, and associate the image details Layer with the key value Key i The similarity calculation is performed to obtain the weight coefficient, and then the weight is multiplied by the original feature, as follows:

[0057]

[0058] Based on the attention mechanism, the multi-scale perception ability of typical feature areas in the image is improved, and then the features are combined with a unified decoding process to generate a fused image.

[0059] In this embodiment, in step S103, the discriminator is a multi-layer network including convolution, and the two discriminators have the same network structure. The discriminator includes 6 convolution modules, each module includes a convolution layer and a LeakyReLU activation function, and the two discriminators respectively check the infrared intensity in the fused image, as well as the depth image details such as distance, contour, and texture.

[0060] In this embodiment, in step S103, the infrared image details include infrared intensity, and the depth image details include distance, contour, and texture.

[0061] In this embodiment, the power equipment includes substation equipment and overhead lines; an infrared thermal imager or an optical zoom infrared imaging device is used to obtain infrared images, and a three-dimensional laser scanner is used to obtain depth images.

[0062] The beneficial effect of the infrared image and depth image fusion method based on the attention mechanism of the embodiment of the present invention is: taking into account that the infrared image and depth image of the power equipment have different modalities, in the image fusion process, the attention mechanism is introduced on the basis of the generator discriminator, and the corresponding features are extracted and image fusion is realized for applications such as non-contact temperature measurement of power equipment. Compared with visible light pictures, the use of depth images can adapt to the difference in image focal length, suppress interference factors such as sunlight and environment, and take into account the requirements of enhancing key areas and retaining equipment detail information, thereby improving the accuracy of equipment status monitoring.

[0063] The above is only a specific implementation of the invention, but the protection scope of the invention is not limited thereto. Those skilled in the art should understand that the invention includes but is not limited to the contents described in the above specific implementation. Any modification that does not deviate from the functional and structural principles of the invention will be included in the scope of the claims.

Claims

1. A method for fusing infrared images and depth images based on an attention mechanism, characterized in that: Applied to a pre-built generative adversarial network, the generative adversarial network includes a generator, an attention feature extraction module and a discriminator, the method includes: Step S101, obtaining an infrared image and a depth image of the electric power equipment, and encoding the infrared image and the depth image of the electric power equipment respectively based on the generator to obtain encoded infrared image and depth image of the electric power equipment; Step S102: Based on the attention feature extraction module, extract attention features from the encoded infrared image and depth image of the power equipment to obtain power infrared features and depth image features, and fuse the power infrared features and the depth image features to obtain a fused image; Step S103: using the discriminator to detect the fused image to obtain infrared image details and depth image details of the fused image; In step S102, the input feature is defined as Feature i , after the global average pooling operation, the key value Key of each feature is obtained i , define the image details to be fused as Layer, and associate the image details Layer with the key value Key i The similarity calculation is performed to obtain the weight coefficient, and then the weight is multiplied by the original feature, as follows: Based on the attention mechanism, the multi-scale perception ability of typical feature areas in the image is improved, and then the features are combined with a unified decoding process to generate a fused image.

2. The infrared image and depth image fusion method based on the attention mechanism according to claim 1, characterized in that: In step S101, the generator includes two data streams, each data stream includes three convolution modules, and each convolution module includes a convolution layer and a ReLU activation function.

3. The infrared image and depth image fusion method based on the attention mechanism according to claim 1, characterized in that: In step S103, the discriminator is a multi-layer network including convolution, there are two discriminators, the two discriminators have the same network structure, and the discriminator includes 6 convolution modules, each module includes a convolution layer and a LeakyReLU activation function.

4. The method for fusion of infrared image and depth image based on attention mechanism according to claim 1, characterized in that: In step S103, the infrared image details include infrared intensity, and the depth image details include at least one of distance, contour, and texture; the power equipment includes substation equipment and overhead lines; an infrared thermal imager or an optical zoom infrared imaging device is used to obtain the infrared image, and a three-dimensional laser scanner is used to obtain the depth image.

5. A system for fusion of infrared image and depth image based on attention mechanism, the system comprising: The encoding module is used to encode the infrared image and the depth image of the electric power equipment respectively to obtain the encoded infrared image and the depth image of the electric power equipment; An attention feature extraction module is used to extract attention features from the encoded infrared image and depth image of the power equipment to obtain power infrared features and depth image features, and fuse the power infrared features and the depth image features to obtain a fused image; A discrimination module, used for detecting the fused image to obtain infrared image details and depth image details of the fused image; The attention feature extraction module defines the input feature as Feature i , after the global average pooling operation, the key value Key of each feature is obtained i , define the image details to be fused as Layer, and associate the image details Layer with the key value Key i The similarity calculation is performed to obtain the weight coefficient, and then the weight is multiplied by the original feature, as follows: Based on the attention mechanism, the multi-scale perception ability of typical feature areas in the image is improved, and then the features are combined with a unified decoding process to generate a fused image.

6. The infrared image and depth image fusion system based on attention mechanism according to claim 5, characterized in that: The encoding module includes two data streams, each of which includes three convolution modules, each of which includes a convolution layer and a ReLU activation function; the discriminant module includes a dual discriminator, the two discriminators have the same network structure, the discriminator is a multi-layer network including convolution, the discriminator includes 6 convolution modules, each module includes a convolution layer and a LeakyReLU activation function, and respectively checks the infrared intensity and depth image details in the fused image, and the depth image details include at least one of distance, contour, and texture.

7. An image fusion device, characterized in that: The image fusion device comprises a memory and a processor connected to each other, the memory stores program instructions, and the processor is used to execute the program instructions to implement the image fusion method according to any one of claims 1 to 4.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store program instructions, and when the program instructions are executed, the image fusion method described in any one of claims 1 to 4 is implemented.

Images