Vision system and method for infrared and visible light fusion based on deep neural network

Through the deep neural network fusion of infrared and visible light images and combining the generation of adversarial network, real-time monitoring and disaster warning of the power system in harsh environments is achieved, and the problems of low inspection efficiency and low data processing efficiency of the power system are solved, which improves the Disaster warning success rate and data processing efficiency.

CN113947555BActive Publication Date: 2024-12-03STATE GRID SHAANXI ELECTRIC POWER CO XIXIAN NEW DISTRICT POWER SUPPLY CO
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202111126389.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-09-26
Publication Date
2024-12-03
Estimated Expiration
2041-09-26

AI Technical Summary

Technical Problem

In harsh environments, the inspection efficiency of the power system is low, traditional manual inspection is time-consuming and labor-intensive, and the data processing efficiency is not high, especially under conditions such as strong winds, strong light and signal interference, which affects the inspection accuracy of the drone and The effectiveness of data transmission.

Method used

Adopt infrared and visible light fusion vision system based on deep neural networks, through image fusion, saliency area temperature measurement and object detection technology, combined with the generation of adversarial network, real-time monitoring of the power system and disaster warning, and improve disaster warning in harsh environments. Success rate.

Benefits of technology

Improve the success rate of disaster warning in harsh environments, improve data processing efficiency, ensure the effectiveness of image acquisition and the accuracy of information refining, realize the monitoring and identification of power systems under extreme weather conditions, and reduce the cost of manpower and material resources. .

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113947555B_ABST
    Figure CN113947555B_ABST
Patent Text Reader

Abstract

The present invention provides a visual method based on deep neural network infrared and visible light fusion, which relates to the technical field of power system monitoring, and includes the following steps: obtaining infrared camera images and visible light camera images through an inspection platform; fusing the infrared camera images and visible light camera images to obtain a fused image; measuring the temperature of a significant area of ​​the fused image; identifying the target and performing a disaster alarm through a target detection technology based on a generative adversarial network. The present invention provides a visual system based on deep neural network infrared and visible light fusion. The present invention fuses the infrared and visible light images according to different fusion weights, and adopts a two-point + neural network + time domain high-pass filter combined with a non-uniform correction algorithm to improve the temperature measurement accuracy, and improves the target recognition accuracy through a target detection technology based on a deep neural network.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of power system monitoring, and in particular to a vision system and method for infrared and visible light fusion based on a deep neural network. Background Art

[0002] With the development of the scale of power systems, the requirements for the safe operation of power lines and the reliability of power supply are getting higher and higher. In order to ensure the normal operation of the power system, power inspection has become one of the important daily tasks of power practitioners. According to the conventional traditional manual inspection method, operation and maintenance personnel need to complete the inspection of various equipment in sequence according to the planned inspection route map, which consumes a lot of manpower and time.

[0003] The workflow of the intelligent unattended inspection image AI processing system usually includes three parts, namely, front-end refined image acquisition, data feedback, and background data processing. How to ensure the effectiveness of image acquisition data and the efficiency of background data processing is an important indicator for measuring the performance of the intelligent unattended power inspection system. During daily inspections, the focus is to quickly discover detailed problems of equipment such as lines and towers, monitor and identify early warnings of the status of the power system, especially disaster early warnings in harsh environments. However, during on-site operations, external forces often have a negative impact on the accuracy of drone inspections. These external forces include: strong winds, strong light, and signal interference.

[0004] In the past, outdoor power inspections were conducted in harsh environments, and power inspectors had to climb mountains and cross ridges, which required a lot of manpower and was not efficient overall. An important part of the inspection process is how to effectively manage all the data collected on site and extract the data into effective information. Data is not the same as information. Data may be massive or large in size, but information must be condensed and summarizable. Under current technical conditions, it is not yet mature to transmit large-scale data samples in real time via wireless links. Summary of the invention

[0005] In order to overcome the problems of the above-mentioned prior art, the present invention provides a vision system and method based on deep neural network for infrared and visible light fusion. Based on the UAV platform, it has multiple payloads such as infrared, visible light and multi-spectrum, which can monitor and identify the status of the power system in real time, greatly improving the success rate of disaster warning in harsh environments.

[0006] The technical solution of the present invention is:

[0007] The vision method of infrared and visible light fusion based on deep neural network includes the following steps:

[0008] Acquire infrared camera images and visible light camera images through the inspection platform;

[0009] Performing image fusion on the infrared camera image and the visible light camera image to obtain a fused image;

[0010] Conduct temperature measurement of significant areas on the fused image;

[0011] Targets are identified and disaster warnings are issued through target detection technology based on generative adversarial networks.

[0012] As a further technical solution of the present invention, the image fusion of the infrared camera image and the visible light camera image to obtain the fused image specifically includes:

[0013] Preprocess the images collected by visible light camera and infrared camera;

[0014] The entropy value of the visible light image is calculated. When the entropy value is higher than the threshold, the visible light image is used as the fused image. When the entropy value is lower than the threshold, the VGG-19 deep neural network is used to complete the fusion of the visible light and infrared images to obtain the fused image.

[0015] As a further technical solution of the present invention, the VGG-19 deep neural network is used to complete the fusion of visible light and infrared images to obtain a fused image, which specifically includes: using the VGG-19 deep neural network to calculate the significant feature maps of images in different bands, so that the final fused image can maintain the significant features of each image, including the following steps:

[0016] Image scale segmentation: Decompose the visible light image into a visible light base layer and a visible light detail layer, and decompose the infrared image into an infrared base layer and an infrared detail layer; specifically includes:

[0017] The original image is divided into two layers, one of which is the base layer and the other is the detail layer. Assuming that the source image is I k , the base image is The detail layer is The base image is obtained by solving the following equation:

[0018]

[0019] Where gx = [-1, 1], gy = [-1, 1]T, they are horizontal and vertical gradient operators, and the detail layer image is equal to the source image minus the base layer image;

[0020]

[0021] Base layer image fusion: base layer fusion of visible light detail layer and infrared base layer; specifically includes:

[0022] The base image contains the common features and redundant information of the active image, and the average weight strategy is used for base image fusion:

[0023]

[0024] Among them, (x, y) represents the position of the corresponding pixel in the image, and α1 and α2 represent different fusion weight values;

[0025] Detail layer image fusion: Detail layer fusion of visible light detail layer and infrared detail layer through deep neural network; specifically including:

[0026] For detail layer images and The deep learning network VGG-19 is used to calculate the fusion weight value, and the deep feature extraction is performed through the VGG-19 network. The weight map is obtained through a multi-layer fusion strategy, and the final detail layer fusion result is obtained by the trained weight value and detail layer image; among them, the multi-layer fusion strategy is as follows:

[0027] Considering that the detail layer image is indicates the feature weight map of the K-th detail layer image, and m is the channel number of the i-th layer;

[0028]

[0029] Φ i () represents a layer i in the VGG network belongs to {1, 2, 3, 4}, representing relu_1_1, relu_2_1, relu_3_1 and relu_4_1; is an M-dimensional vector;

[0030] The L1 norm of can be used as the activity level evaluation function of the detail layer of the source image, so the initial active layer weight map can be obtained by the following formula:

[0031]

[0032] The final activity weight image is calculated using the block-based average operator It can be obtained by the following formula:

[0033]

[0034] Among them, γ represents the size of the block;

[0035] When the active layer image is obtained The initial weight map is obtained through the soft-max function:

[0036]

[0037] K represents the number of active layer images, set K = 2, It indicates that the initial weight map is in the range of (0-1); it is known that the pooling layer operator of the VGG network is a downsampling algorithm. Each time this operator will reduce the image to 1 / s times of the original, let s=2, so the span of the pooling layer is set to 2 in the VGG network, so the size of the weight map at different layers is 1 / 2i-1 times of the detail layer image, i belongs to {1, 2, 3, 4}, representing the four layers of relu_1_1, relu_2_1, relu_3_1 and relu_4_1; after obtaining the initial weight map, the weight map is upsampled, and the small-sized weight map is enlarged to an image of the same size as the detail layer, and four pairs of weight images are obtained. For each pair of weight maps, the result of detail layer fusion is shown as follows:

[0038]

[0039] Finally, the detail layer image fusion is obtained by the following formula:

[0040]

[0041] F d The meaning of the function is to take the maximum value of each layer as the final fused image of the detail layer;

[0042] Final image fusion: The base layer fusion and detail layer fusion are combined to form a fused image; specifically including:

[0043] When the base layer image and detail layer image have been fused, we add them together using the following formula to get the final fused image:

[0044] F(x, y) = F b (x,y)+F d (x, y);

[0045] where F b represents the fused image of the base layer, F d A fused image representing the detail layer.

[0046] Furthermore, the value of γ is 1.

[0047] Furthermore, the α1 and α2 are both equal to 0.5.

[0048] As a further technical solution of the present invention, the temperature of significant areas of the fused image is measured; specifically, the following steps are performed: extracting significant areas from the fused image through superpixel segmentation, measuring and screening the temperature of the significant areas, issuing an early warning and sending an early warning message for significant areas where the temperature is greater than a set value; and at the same time, smoke detection based on a deep learning model is performed on significant areas where the temperature is greater than the set value to determine whether a fire has occurred in the significant areas where the temperature is greater than the set value.

[0049] As a further technical solution of the present invention, the smoke detection based on the deep learning model is performed on the area within the image range to determine whether a fire occurs in the area, specifically including:

[0050] A deep learning-based model is used to train the characteristics of smoke generated during a fire;

[0051] Generative adversarial networks are used to conduct preliminary training for smoke detection, and the trained weight values ​​are input into the early warning system as judgment parameters;

[0052] When smoke generated by a fire is detected, it is uploaded to the early warning terminal via a wireless network to sound an alarm, indicating that a fire has occurred in the area.

[0053] As a further technical solution of the present invention, the fused image is segmented by superpixel to extract significant areas, the temperature of the significant areas is measured and screened, and a warning is issued and a warning message is sent to the significant areas whose temperature is greater than a set value; specifically, the temperature measurement of the significant areas is performed by using a two-point + neural network + time domain high-pass filtering combined with a non-uniform correction algorithm, and a colorimetric temperature measurement temperature and ratio inversion high-order fitting algorithm to measure the temperature;

[0054] Among them, based on the neural network time domain high-pass filtering combined with the non-uniform correction algorithm, the high and low temperature blackbody images are first collected, and the correction gain and offset coefficients are calculated according to the two-point method, and used as the initial value of the correction coefficient of the improved neural network method. The calculation formula of the two-point method is as follows:

[0055]

[0056] Among them, the response output of high-temperature blackbody and low-temperature blackbody images are respectively expressed as x(T H ) and x(T L ) indicates that the average output responses are and G and O represent the gain factor and offset factor respectively;

[0057] On the basis of the two-point method, the neural network method is used to take the neighborhood weighted average of the output response as the expected output value of the iterative algorithm. Through repeated iterations, the correction gain and offset coefficient converge. The calculation formula of the neural network method is as follows:

[0058]

[0059] where f n (i,j) is the output expected value expression, G n+1 (i, j) and O n+1 (i, j) are the iteration gain coefficient and offset coefficient respectively;

[0060] Based on the neural network method, the time domain high-pass filtering algorithm is used to complete the update of the offset coefficient without changing the gain coefficient. The calculation formula of the time domain high-pass filtering algorithm is as follows:

[0061]

[0062] y(n)=x(n)-f(n);

[0063] Where x(n) represents the nth frame image output from the focal plane, f(n) represents the low-frequency part of the input image, and y(n) represents the corrected output image;

[0064] Among them, the colorimetric temperature measurement technology determines the true temperature of the target by establishing a functional relationship between the ratio of the spectral radiation brightness of the target at two wavelengths and the temperature; according to Planck's law of blackbody radiation, the spectral radiance of the actual object can be expressed as the following formula:

[0065]

[0066] Where M is the spectral radiance and T is the temperature of the object;

[0067] When the temperature of the object is T, the ratio of the radiance corresponding to the central wavelengths λ1 and λ2 is:

[0068]

[0069] Where R is the ratio of radiance;

[0070] It can be seen from the above formula that for the colorimetric temperature measurement system, T and R are in a single-value relationship, and T can be obtained as long as R is obtained;

[0071] In order to improve the accuracy of colorimetric temperature measurement, a high-order fitting algorithm is used to obtain the fitting coefficients of T and R. The calculation relationship between T and R is as follows:

[0072]

[0073] By calculation:

[0074]

[0075] The following linear equations can be obtained:

[0076]

[0077] For a0, a1, ..., a n By solving the linear equations, we can get the polynomial fitting coefficients of T and R.

[0078] As a further technical solution of the present invention, the target is identified and a disaster alarm is performed by using a target detection technology based on a generative adversarial network, specifically including:

[0079] The generative adversarial network consists of two models, the generative model G and the discriminative model D. The random noise z is generated by G to generate samples G(z) that try to obey the real data distribution Pdata. The discriminative model D can determine whether the input sample is the real data x or the generated data G(z). G and D are nonlinear mapping functions.

[0080] The specific algorithm of the generative adversarial network is as follows: First, given the generator, optimize the discriminator; the discriminator is a two-class model, and training the discriminator is the process of minimizing the cross entropy; E(·) is the calculation of the expected value, x is sampled from the real data distribution Pdata(x), and z is sampled from the prior distribution Pz(z); in order to learn the distribution of data x, the generator constructs a mapping space G(z; θ) from the prior noise distribution Pz(z) z ), the corresponding discriminator mapping function is D(x; θ d ), output a scalar indicating the probability that x is the real data;

[0081]

[0082] The loss function of the generator is defined as the cross entropy between the output of the generator and "1"; the loss function of the discriminator consists of two parts: 1) the real sample crosses the entropy between the output of the discriminator and "1"; the sample generated by the generator crosses the entropy between the output of the discriminator and "0"; the loss function of the discriminator is the sum of the two parts; after obtaining the loss functions of the discriminator and the generator, select the Adam optimizer to optimize the loss function.

[0083] The vision system based on deep neural network infrared and visible light fusion includes:

[0084] A visible light and infrared image acquisition unit acquires infrared camera images and visible light camera images through the inspection platform;

[0085] An image fusion unit, fusing the infrared camera image and the visible light camera image to obtain a fused image;

[0086] The temperature measurement unit measures the temperature of the significant area of ​​the fused image;

[0087] The identification and warning unit identifies the target and issues disaster warnings through target detection technology based on generative adversarial networks.

[0088] Furthermore, the visible light and infrared image unit is a drone inspection platform.

[0089] The beneficial effects of the present invention are:

[0090] 1. The present invention proposes a method based on a deep learning algorithm to calculate different fusion weights of infrared and visible light images, and then fuse the images according to different weights to combine the important information in the two images into one image;

[0091] 2. The present invention adopts two-point + neural network + time domain high-pass filtering combined with non-uniform correction algorithm, as well as colorimetric temperature measurement temperature and ratio inversion high-order fitting algorithm to improve temperature measurement accuracy and ensure that the temperature measurement accuracy reaches 1%;

[0092] 3. The present invention uses the target detection technology based on deep neural network, and uses the adversarial network model. The generator fits the data generation process to generate model samples. The optimization goal is to achieve Nash equilibrium so that the generator estimates the distribution of data samples. GAN network has been widely studied and applied in the field of image and vision. Because it has two parts, generator and discriminator, it can continuously improve the recognition ability of the target during training, and the target recognition accuracy is higher. BRIEF DESCRIPTION OF THE DRAWINGS

[0093] Figure 1 This is a flow chart of the visual method based on deep neural network infrared and visible light fusion proposed by the present invention;

[0094] Figure 2 This is the image fusion framework diagram proposed by the present invention;

[0095] Figure 3 This is a detail layer image fusion structure diagram proposed by the present invention;

[0096] Figure 4 The adversarial generation network flow chart proposed by the present invention;

[0097] Figure 5 This is a structural diagram of the visual system proposed in the present invention that combines infrared and visible light based on a deep neural network. DETAILED DESCRIPTION

[0098] The following will be combined with the embodiments and drawings to clearly and completely describe the concept, specific structure and technical effects of the present invention, so as to fully understand the purpose, characteristics and effects of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, other embodiments obtained by those skilled in the art without creative work are all within the scope of protection of the present invention.

[0099] This project uses laser lighting technology, infrared and visible light fusion technology, and artificial intelligence technologies to overcome interference from extreme weather conditions such as fog, rain, snow, and haze, and to understand the extent of damage to the power site at the first possible moment.

[0100] See also Figure 1 , which is a flow chart of the visual method based on deep neural network infrared and visible light fusion proposed by the present invention;

[0101] like Figure 1 As shown, the vision method of infrared and visible light fusion based on deep neural network includes the following steps:

[0102] Step 101, acquiring an infrared camera image and a visible light camera image through an inspection platform;

[0103] Step 102, performing image fusion on the infrared camera image and the visible light camera image to obtain a fused image;

[0104] Step 103, measuring the temperature of a significant area of ​​the fused image;

[0105] Step 104: Identify the target and issue a disaster warning using a target detection technology based on a generative adversarial network.

[0106] In the embodiment of the present invention, the infrared camera image and the visible light camera image are fused to obtain a fused image; specifically, the method includes:

[0107] Preprocess the images collected by visible light camera and infrared camera;

[0108] The entropy value of the visible light image is calculated. When the entropy value is higher than the threshold, the visible light image is used as the fused image. When the entropy value is lower than the threshold, the VGG-19 deep neural network is used to complete the fusion of the visible light and infrared images to obtain the fused image.

[0109] First, the visible light camera and infrared camera are preprocessed, including image enhancement, denoising and other processing. Then the entropy value of the visible light image is calculated. When the entropy value is higher than the threshold, we can think that it is currently in the daytime stage, and the visible light camera will not be fused with the infrared image. When the entropy value is lower than the threshold, we can think that it is currently in the night stage. Since the infrared image cannot well reflect the texture and detail information of the detected object, the visible light camera is turned on to fuse with the infrared image, which can ensure that the fused image always has better details and texture information. Then the VGG-19 deep neural network is used to complete the fusion of visible light and infrared images;

[0110] See also Figure 2 In an embodiment of the present invention, a VGG-19 deep neural network is used to complete the fusion of visible light and infrared images to obtain a fused image, which specifically includes: using the VGG-19 deep neural network to calculate the significant feature maps of images in different bands, so that the final fused image can maintain the significant features of each image, including the following steps:

[0111] Image scale segmentation: Decompose the visible light image into a visible light base layer and a visible light detail layer, and decompose the infrared image into an infrared base layer and an infrared detail layer; specifically includes:

[0112] The original image is divided into two layers, one of which is the base layer and the other is the detail layer. Assuming that the source image is I k , the base image is The detail layer is The base image is obtained by solving the following equation:

[0113]

[0114] Where gx = [-1, 1], gy = [-1, 1]T, they are horizontal and vertical gradient operators, and the detail layer image is equal to the source image minus the base layer image;

[0115]

[0116] Base layer image fusion: base layer fusion of visible light detail layer and infrared base layer; specifically includes:

[0117] The base image contains the common features and redundant information of the active image, and the average weight strategy is used for base image fusion:

[0118]

[0119] Among them, (x, y) represents the position of the corresponding pixel in the image, and α1 and α2 represent different fusion weight values;

[0120] See also Figure 3 , Detail layer image fusion: Detail layer fusion of visible light detail layer and infrared detail layer through deep neural network; specifically including:

[0121] For detail layer images and The deep learning network VGG-19 is used to calculate the fusion weight value, and the deep feature extraction is performed through the VGG-19 network. The weight map is obtained through a multi-layer fusion strategy, and the final detail layer fusion result is obtained by the trained weight value and detail layer image; among them, the multi-layer fusion strategy is as follows:

[0122] Considering that the detail layer image is indicates the feature weight map of the K-th detail layer image, and m is the channel number of the i-th layer;

[0123]

[0124] Φ i() represents a layer i in the VGG network belongs to {1, 2, 3, 4}, representing relu_1_1, relu_2_1, relu_3_1 and relu_4_1; is an M-dimensional vector;

[0125] The L1 norm of can be used as the activity level evaluation function of the detail layer of the source image, so the initial active layer weight map can be obtained by the following formula:

[0126]

[0127] The final activity weight image is calculated using the block-based average operator It can be obtained by the following formula:

[0128]

[0129] Among them, γ represents the size of the block;

[0130] When the active layer image is obtained The initial weight map is obtained through the soft-max function:

[0131]

[0132] K represents the number of active layer images, set K = 2, It indicates that the initial weight map is in the range of (0-1); it is known that the pooling layer operator of the VGG network is a downsampling algorithm. Each time this operator will reduce the image to 1 / s times of the original, let s=2, so the span of the pooling layer is set to 2 in the VGG network, so the size of the weight map at different layers is 1 / 2i-1 times of the detail layer image, i belongs to {1, 2, 3, 4}, representing the four layers of relu_1_1, relu_2_1, relu_3_1 and relu_4_1; after obtaining the initial weight map, the weight map is upsampled, and the small-sized weight map is enlarged to an image of the same size as the detail layer, and four pairs of weight images are obtained. For each pair of weight maps, the result of detail layer fusion is shown as follows:

[0133]

[0134] Finally, the detail layer image fusion is obtained by the following formula:

[0135]

[0136] F d The meaning of the function is to take the maximum value of each layer as the final fused image of the detail layer;

[0137] Final image fusion: The base layer fusion and detail layer fusion are combined to form a fused image; specifically including:

[0138] When the base layer image and detail layer image have been fused, we add them together using the following formula to get the final fused image:

[0139] F(x,y)=F b (x,y)+F d (x,y);

[0140] where F b represents the fused image of the base layer, F d A fused image representing the detail layer.

[0141] Furthermore, the value of γ is 1. If γ becomes larger, the fused image will become more dynamic, but image details will be easily lost, so γ=1 is selected.

[0142] Furthermore, α1 and α2 are both equal to 0.5. In order to retain more general information and weaken redundant information, we choose α1 and α2 to be equal to 0.5.

[0143] In an embodiment of the present invention, the technology of infrared and visible light multi-source information fusion based on deep neural network specifically includes: first, decomposing the image into a base layer image and a detail layer image, and the base layer image is fused using an average weight algorithm; secondly, for the detail layer, we use the VGG-19 multi-layer neural network to extract multi-layer features, and then use the L1 norm and average weight to generate candidate images of the detail fusion image, and use the maximum candidate image as the final detail layer fusion image; finally, the base layer image and the detail layer image are added to form the final fused image.

[0144] The present invention proposes a deep learning algorithm to calculate different fusion weights of infrared and visible light images, and then perform image fusion according to the different weights.

[0145] Furthermore, the temperature of significant areas of the fused image is measured; specifically, the method includes: extracting significant areas of the fused image through superpixel segmentation, measuring and screening the temperature of the significant areas, issuing warnings and sending warning information for significant areas where the temperature is greater than a set value; and at the same time, performing smoke detection based on a deep learning model for significant areas where the temperature is greater than the set value to determine whether a fire has occurred in the significant areas where the temperature is greater than the set value.

[0146] The fused image is segmented using superpixel images to extract significant areas, and the temperature of the significant areas is measured. Usually, significant areas include power equipment, wires and other areas prone to fire hazards. When it is found that the temperature of a certain area is much higher than that of other areas, the system will send information to the early warning system terminal via the wireless network, indicating that the area has a fire hazard.

[0147] While detecting the temperature of the significant area, smoke detection is performed on the area within the field of view of the image. Since there are many types of smoke, low-concentration smoke does not mean that a fire has occurred. Therefore, in order to reduce the misjudgment rate of fire, we use a deep learning-based model to train the characteristics of smoke generated during a fire. We use a generative adversarial network to perform preliminary training for smoke detection, and input the trained weight values ​​as judgment parameters into the early warning system. When the system detects smoke generated by a fire, it uploads it to the early warning terminal through a wireless network to issue an alarm, indicating that a fire has occurred in the area.

[0148] Furthermore, the smoke detection based on the deep learning model is performed on the area within the image range to determine whether a fire occurs in the area, specifically including:

[0149] A deep learning-based model is used to train the characteristics of smoke generated during a fire;

[0150] Generative adversarial networks are used to conduct preliminary training for smoke detection, and the trained weight values ​​are input into the early warning system as judgment parameters;

[0151] When smoke generated by a fire is detected, it is uploaded to the early warning terminal via a wireless network to sound an alarm, indicating that a fire has occurred in the area.

[0152] Furthermore, the fused image is segmented by superpixel to extract significant areas, the temperature of the significant areas is measured and screened, and a warning is issued and a warning message is sent to the significant areas where the temperature is greater than a set value; specifically, the temperature measurement of the significant areas is performed by using a two-point + neural network + time domain high-pass filter combined with a non-uniform correction algorithm, and a colorimetric temperature measurement temperature and ratio inversion high-order fitting algorithm to measure the temperature;

[0153] Among them, based on the neural network time domain high-pass filtering combined with the non-uniform correction algorithm, the high and low temperature blackbody images are first collected, and the correction gain and offset coefficients are calculated according to the two-point method, and used as the initial value of the correction coefficient of the improved neural network method. The calculation formula of the two-point method is as follows:

[0154]

[0155]

[0156] Among them, the response output of high-temperature blackbody and low-temperature blackbody images are respectively expressed as x(T H ) and x(T L ) indicates that the average output responses are and G and O represent the gain factor and offset factor respectively;

[0157] On the basis of the two-point method, the neural network method is used to take the neighborhood weighted average of the output response as the expected output value of the iterative algorithm. Through repeated iterations, the correction gain and offset coefficient converge. The calculation formula of the neural network method is as follows:

[0158]

[0159] where f n (i,j) is the output expected value expression, G n+1 (i, j) and O n+1 (i, j) are the iteration gain coefficient and offset coefficient respectively;

[0160] Based on the neural network method, the time domain high-pass filtering algorithm is used to complete the update of the offset coefficient without changing the gain coefficient. The calculation formula of the time domain high-pass filtering algorithm is as follows:

[0161]

[0162] y(n)=x(n)-f(n);

[0163] Where x(n) represents the nth frame image output from the focal plane, f(n) represents the low-frequency part of the input image, and y(n) represents the corrected output image;

[0164] In the embodiment of the present invention, two points + neural network + time domain high-pass filtering are combined with a non-uniform correction algorithm, and the output image has a high contrast and clear image edges. The algorithm converges in a short time, can remove overly dark areas, and suppress fixed pattern noise.

[0165] Among them, the colorimetric temperature measurement technology determines the true temperature of the target by establishing a functional relationship between the ratio of the spectral radiation brightness of the target at two wavelengths and the temperature; according to Planck's law of blackbody radiation, the spectral radiance of the actual object can be expressed as the following formula:

[0166]

[0167] Where M is the spectral radiance and T is the temperature of the object;

[0168] When the temperature of the object is T, the ratio of the radiance corresponding to the central wavelengths λ1 and λ2 is:

[0169]

[0170] Where R is the ratio of radiance;

[0171] It can be seen from the above formula that for the colorimetric temperature measurement system, T and R are in a single-value relationship, and T can be obtained as long as R is obtained;

[0172] In order to improve the accuracy of colorimetric temperature measurement, a high-order fitting algorithm is used to obtain the fitting coefficients of T and R. The calculation relationship between T and R is as follows:

[0173]

[0174] By calculation:

[0175]

[0176] The following linear equations can be obtained:

[0177]

[0178] For a0, a1, ..., a n By solving the linear equations, we can get the polynomial fitting coefficients of T and R.

[0179] In the embodiment of the present invention, a two-point + neural network + time domain high-pass filtering combined with a non-uniform correction algorithm and a colorimetric temperature measurement temperature and ratio inversion high-order fitting algorithm are used to improve the temperature measurement accuracy and ensure that the temperature measurement accuracy reaches 1%.

[0180] See also Figure 4 In the implementation of the present invention, the target is identified and a disaster alarm is performed by using a target detection technology based on a generative adversarial network, which specifically includes:

[0181] The generative adversarial network consists of two models, the generative model G and the discriminative model D. The random noise z is generated by G to generate samples G(z) that try to obey the real data distribution Pdata. The discriminative model D can determine whether the input sample is the real data x or the generated data G(z). G and D are nonlinear mapping functions, such as multi-layer perceptrons.

[0182] The specific algorithm of the generative adversarial network is as follows: First, given the generator, optimize the discriminator; the discriminator is a two-class model, and training the discriminator is the process of minimizing the cross entropy; E(·) is the calculation of the expected value, x is sampled from the real data distribution Pdata(x), and z is sampled from the prior distribution Pz(z); in order to learn the distribution of data x, the generator constructs a mapping space G(z; θ) from the prior noise distribution Pz(z) z ), the corresponding discriminator mapping function is D(x; θ d ), output a scalar indicating the probability that x is the real data;

[0183]

[0184] The goal of the generator is to make the output of the generator close to 1 after passing through the discriminator (the generated sample is close to the real sample); and the purpose of the discriminator is, on the one hand, to make the result of the real sample close to 1 after passing through the discriminator, and on the other hand, to make the result of the sample generated by the generator close to 0 after passing through the discriminator.

[0185] The loss function of the generator is defined as the cross entropy between the output of the generator and "1"; the loss function of the discriminator consists of two parts: the real sample crosses the entropy between the output of the discriminator and "1"; the sample generated by the generator crosses the entropy between the output of the discriminator and "0"; the loss function of the discriminator is the sum of the two parts; after obtaining the loss functions of the discriminator and the generator, the Adam optimizer is selected to optimize the loss function. In the actual training process, the discriminator will easily win in the adversarial training with the generator, resulting in the vanishing gradient of the generator. Therefore, in training, updating the discriminator once requires updating the generator k (k>1) times to prevent the discriminator from quickly reaching (approximate) optimality during the training process, so as to maintain the adversarial balance between the generator and the discriminator; the choice of k value should also be selected according to different scales of data sets. If the k value is too small, the discriminator will reach (approximate) optimality, and the generator will have vanishing gradients and the loss function will not be reduced; if the k value is too large, the gradient of the generator will be inaccurate and oscillate back and forth. The trained GAN is applied to target detection and recognition; the trained GAN discriminator part (removing the last layer) is extracted and the parameters are fine-tuned in the new structure. Unlike the discriminator structure, the output dimension of the last fully connected layer is n; the output result is passed through the Softmax classifier; the loss function obtained by cross entropy with the label y is also optimized using Adam; in addition, when training this structure, the Dropout value is set to 0.5 to prevent overfitting of the fully connected layer.

[0186] Generative adversarial network (GAN) is a generative model. Different from traditional generative models, its network structure contains a discriminative network in addition to the generative network. The relationship between the generative network and the discriminative network is an adversarial one. The idea of ​​adversarial comes from game theory. In an equal game, the two sides of the game use each other's strategy to change their own adversarial strategy in order to win. In generative adversarial networks, the generator and the discriminator are the two sides of the game. The generator fits the data generation process to generate model samples. The optimization goal is to achieve Nash equilibrium so that the generator estimates the distribution of data samples. GAN networks have been widely studied and applied in the fields of image and vision. Because they have two parts, the generator and the discriminator, they can continuously improve the recognition ability of the target during training. Therefore, after the training is completed, the target recognition accuracy is higher.

[0187] Compared with the general supervised learning method for image classification, all feature extraction layers in the classification model used in this method will be pre-trained in the dataset as the discriminator of GAN. Its advantage is that in addition to the real samples in the dataset, the samples generated by the GAN generator will also be input into the discriminator as data, which plays a role in data enhancement, so that the discriminator can better recognize ground targets through training.

[0188] See also Figure 5 , which is a structural diagram of the visual system based on deep neural network infrared and visible light fusion proposed by the present invention;

[0189] like Figure 5 As shown in the figure, the vision system based on deep neural network infrared and visible light fusion includes:

[0190] A visible light and infrared image acquisition unit 201 acquires infrared camera images and visible light camera images through an inspection platform;

[0191] An image fusion unit 202 performs image fusion on the infrared camera image and the visible light camera image to obtain a fused image;

[0192] The temperature measuring unit 203 measures the temperature of a significant area of ​​the fused image;

[0193] The identification and warning unit 204 identifies the target and issues a disaster warning through a target detection technology based on a generative adversarial network.

[0194] The long-distance vision system based on deep neural network and infrared and visible light multi-source information fusion can be a handheld or drone-mounted detection system.

[0195] The present invention can be widely used to achieve visualization of specific targets in various severe weather conditions. The system can understand the real-time situation in a specific scene under any extreme conditions, and provide 24-hour uninterrupted images for power facility inspections, disaster area conditions and other specific targets. It can solve the problems of real-time monitoring of specific targets in various extreme weather conditions, especially at night, timely troubleshooting and handling of safety hazards, and comprehensive understanding of disasters at the first time, while saving a lot of manpower and material resources. Its application in the power industry will significantly improve social and economic benefits, and can also avoid other secondary disasters caused by hazards at night, causing large-scale power outages and economic losses.

[0196] In the power industry, it can be promoted and applied in various extreme conditions, such as heavy rain, heavy fog, thick smoke, haze and other weather conditions to achieve long-distance observation of power equipment. In particular, it can achieve long-distance, high-definition observation of specific targets such as disaster risk points, disaster areas, power facilities, etc. at night without any natural light.

[0197] The embodiments of the present invention save manpower and material costs. It is possible to monitor specific power equipment at a distance at any time and in any weather conditions, without the need for manual inspection by operation and maintenance personnel. When the system finds an abnormality, it can promptly notify the operation and maintenance personnel to handle it. Disaster prevention. Potential hazards can be monitored in real time 24 hours a day, and effective measures can be taken when a disaster occurs to avoid secondary disasters. Disaster area monitoring can determine the disaster situation in the disaster area that has already occurred, and prepare accurate disaster relief supplies based on the survey situation.

[0198] The present invention is described in detail above, but the present invention is not limited to the above-mentioned embodiments. Various changes can be made within the knowledge of ordinary technicians in the field without departing from the purpose of the present invention. Many other changes and modifications can be made without departing from the concept and scope of the present invention. It should be understood that the present invention is not limited to specific embodiments, and the scope of the present invention is defined by the appended claims.

Claims

1. A visual method based on deep neural network for infrared and visible light fusion, characterized in that: The following steps are involved: Acquire infrared camera images and visible light camera images through the inspection platform; Performing image fusion on the infrared camera image and the visible light camera image to obtain a fused image; Conduct temperature measurement of significant areas on the fused image; Targets are identified and disaster warnings are issued through target detection technology based on generative adversarial networks; The target detection technology based on the generative adversarial network is used to identify the target and issue a disaster warning, specifically including: The generative adversarial network consists of two models, the generative model G and the discriminative model D. The random noise z is generated by G to generate samples G(z) that try to obey the real data distribution Pdata. The discriminative model D can determine whether the input sample is the real data x or the generated data G(z). G and D are nonlinear mapping functions. The specific algorithm of the generative adversarial network is as follows: First, given the generator, optimize the discriminator; the discriminator is a two-class model, and training the discriminator is the process of minimizing the cross entropy; E(·) is the calculation of the expected value, x is sampled from the real data distribution Pdata(x), and z is sampled from the prior distribution Pz(z); in order to learn the distribution of data x, the generator constructs a mapping space G(z; θ) from the prior noise distribution Pz(z) z ), the corresponding discriminator mapping function is D(x; θ d ), output a scalar indicating the probability that x is the real data; The loss function of the generator is defined as the cross entropy between the output of the generator and "1". The loss function of the discriminator consists of two parts: the real sample crosses the entropy between the output of the discriminator and "1"; the sample generated by the generator crosses the entropy between the output of the discriminator and "0". The loss function of the discriminator is the sum of the two parts. After obtaining the loss functions of the discriminator and the generator, select the Adam optimizer to optimize the loss function.

2. The method according to claim 1, characterized in that The step of fusing the infrared camera image and the visible light camera image to obtain a fused image specifically includes: Preprocess the images collected by visible light camera and infrared camera; The entropy value of the visible light image is calculated. When the entropy value is higher than the threshold, the visible light image is used as the fused image. When the entropy value is lower than the threshold, the VGG-19 deep neural network is used to complete the fusion of the visible light and infrared images to obtain the fused image.

3. The method according to claim 1, characterized in that The method of measuring the temperature of a significant area of ​​the fused image specifically includes: extracting a significant area from the fused image through superpixel segmentation, measuring and screening the temperature of the significant area, issuing an early warning and sending an early warning message to the significant area where the temperature is greater than a set value; and at the same time, performing smoke detection based on a deep learning model on the significant area where the temperature is greater than the set value to determine whether a fire occurs in the significant area where the temperature is greater than the set value.

4. The method according to claim 3, characterized in that The significant area greater than the set value adopts smoke detection based on the deep learning model to determine whether a fire occurs in the significant area greater than the set value, specifically including: A deep learning-based model is used to train the characteristics of smoke generated during a fire; Generative adversarial networks are used to conduct preliminary training for smoke detection, and the trained weight values ​​are input into the early warning system as judgment parameters; When smoke generated by a fire is detected, it is uploaded to the early warning terminal via a wireless network to sound an alarm, indicating that a fire has occurred in the area.

5. A vision system based on deep neural network infrared and visible light fusion, using the vision method based on deep neural network infrared and visible light fusion as described in any one of claims 1 to 4, characterized in that: include: A visible light and infrared image acquisition unit acquires infrared camera images and visible light camera images through the inspection platform; An image fusion unit, fusing the infrared camera image and the visible light camera image to obtain a fused image; The temperature measurement unit measures the temperature of the significant area of ​​the fused image; The identification and warning unit identifies the target and issues disaster warnings through target detection technology based on generative adversarial networks.