Hardware heating infrared image data augmentation method based on deep learning

Abstract

The invention provides a hardware fitting heating infrared image data augmentation method based on deep learning, and relates to the technical field of visual inspection, and the method comprises the following steps: collecting a hardware fitting infrared image, building a hardware fitting heating infrared image data set, building a deep convolutional neural network model, training a generative adversarial network model, and generating an infrared image sample. According to the method, infrared sample images of different styles are generated by adopting a deep learning method, so that the number of samples is increased, and the diversity of sample features can be effectively increased. According to the method, the generator is improved by means of cross-layer connection, dense residual block connection, introduction of an attention mechanism into residual blocks, introduction of a spatial pyramid pooling module and the like, partial performance of the generator in the model is improved under the condition that the complexity of the model is not greatly increased, semantic information of different scales is extracted, and the accuracy of the model is improved. And a better feature extraction capability is obtained.

Description

Technical Field

[0001] The present invention relates to the field of visual inspection technology, and in particular to a method for augmenting hardware heating infrared image data based on deep learning. Background Art

[0002] With the rapid development of transmission lines, the domestic economy's demand for electricity continues to increase. The safe operation of transmission lines will be an important guarantee for domestic economic construction. It is urgent to improve the identification of fault hazards in transmission lines. Hardware is used to connect or reinforce insulators, power lines and other power equipment to transfer electrical loads, mechanical loads and play a protective role. Due to the wide variety of hardware, the main hardware power equipment includes anti-vibration hammers, voltage equalizing rings, spacers, connecting plates, bolts, pins, etc. Transmission lines contain many kinds of hardware. Since transmission lines are erected in various natural environments, they are exposed to wind, sun and rain all year round, and are sometimes affected by severe weather such as ice and snow, which will inevitably cause power equipment to be missing or damaged. According to the management regulations of (State Grid (Operation Inspection / 4) 305-2014), overhead line equipment contains 878 kinds of defects, and common hardware defects in line inspections account for 111 kinds.

[0003] Regular power inspection is a basic work that can effectively ensure the safe and orderly operation of power transmission lines and power transmission equipment. The traditional manual inspection method not only consumes a lot of manpower and material resources, but also has high risks. Helicopters and drones equipped with infrared thermal imagers have become the mainstream inspection method. Infrared thermal imaging can safely and quickly capture the hot spots in the transmission lines, which provides great convenience for power inspection. However, with the continuous development and construction of my country's power grid, the scale of the power grid has also increased. The amount of infrared spectrum data collected by the infrared thermal imager of the inspection robot has increased exponentially. How to process massive infrared image data in time to quickly and effectively diagnose power equipment with infrared has become a difficult problem faced by operation and inspection staff. Traditional manual or algorithmic identification of hot spots in equipment and instruments has low efficiency and poor data accuracy, which can no longer meet the construction and development needs of rapidly developing transmission lines. Based on infrared image data, relying on artificial intelligence technology to realize the full-process intelligent processing of various scene data, it can quickly find line defects and hidden dangers and improve the lean operation and maintenance level of the power grid. However, the research on hardware heating target detection based on deep learning requires a large number of infrared images. The existing hardware heating infrared image data is scarce and difficult to obtain, which affects the intelligent inspection of transmission lines. Summary of the invention

[0004] 1. Technical issues to be resolved

[0005] In view of the problems that infrared images of heating fittings in transmission lines are scarce and difficult to obtain, the present invention provides a data augmentation method for infrared images of heating fittings based on deep learning. The CycleGAN network model is used to generate infrared sample images of different styles, which not only expands the number of samples, but also effectively amplifies the diversity of sample features.

[0006] (II) Technical solution

[0007] To achieve the above objectives, the present invention is implemented through the following technical solutions:

[0008] A method for augmenting hardware heating infrared image data based on deep learning, comprising the following steps:

[0009] S1. Collect infrared images of hardware: Use a drone equipped with infrared cameras to detect the status of power transmission line hardware. The collected inspection images are stored in a memory card or wirelessly transmitted to a local server.

[0010] S2. Establishing a hardware heating infrared image dataset: Select the hardware heating infrared image in step S1 and adjust the image to a uniform size of 256×256;

[0011] S3. Build a deep convolutional neural network model: Use Python language to build a generative adversarial network model under the pytorch deep learning framework;

[0012] S4, training a generative adversarial network model: using transfer learning and the hardware heating infrared image dataset established in step S2 to train and optimize the deep convolutional neural network model built in step S3, to obtain a generative adversarial network model;

[0013] S5. Generate infrared image samples: Select the heating infrared image of the hardware and input it into the generative adversarial network model trained in step S4 to perform infrared image prediction, thereby completing the infrared image sample expansion.

[0014] Furthermore, the hardware in the data set of step S2 includes tension clamps and suspension clamps.

[0015] Furthermore, the generative adversarial network model in step S3 is a CycleGAN network model, which includes a first generator G, a first discriminator D X , second generator F, second discriminator D YThe hardware heating infrared image data set is divided into X-domain images and Y-domain images, the X-domain images contain real samples X, and the Y-domain images contain real samples Y; the first generator G generates generated samples G(X) and reconstructed samples G(F(Y)), and the second generator F generates generated samples F(Y) and reconstructed samples F(G(X)); the real samples X and generated samples F(Y) are connected to the first generator G to generate generated samples G(X), and the generated samples G(X) and the real samples Y are connected to the second discriminator D Y It is used to judge the authenticity of the generated samples and obtain the loss function L GAN2 The real sample Y and the generated sample G(X) are connected to the second generator F to generate the generated sample F(Y), and the generated sample F(Y) is connected to the real sample X and the first discriminator D X It is used to judge the authenticity of the generated samples and obtain the loss function L GAN1 The reconstructed sample F(G(X)) and the real sample X, the reconstructed sample G(F(Y)) and the real sample Y constitute a cyclic loss function L cyc .

[0016] Furthermore, the first generator G and the second generator F include an encoding part, a feature extraction part, and a decoding part. The encoding part reduces the image size by downsampling, the feature extraction part is used to extract the heating image features of the hardware, and the decoding part restores the image size by interpolation bilinear upsampling. The encoding part, the feature extraction part, and the decoding part are connected in sequence; the encoding part includes four convolutional layers, namely, the first convolutional layer, the second convolutional layer, the third convolutional layer, and the fourth convolutional layer. The feature extraction part is composed of 8 residual blocks connected in sequence, and the decoding part includes four deconvolutional layers, namely, the first deconvolutional layer, the second deconvolutional layer, the third deconvolutional layer, and the fourth deconvolutional layer; the encoding part and the decoding part are also connected in a cross-layer manner, that is, the output of the first convolutional layer is also connected to the input of the first deconvolutional layer, the output of the second convolutional layer is also connected to the input of the second deconvolutional layer, the output of the third convolutional layer is also connected to the input of the third deconvolutional layer, and the output of the fourth convolutional layer is also connected to the input of the fourth deconvolutional layer.

[0017] Furthermore, the residual block includes a 3*3 convolutional layer, an activation layer, a 1*1 convolutional layer, a maximum pooling layer, an average pooling layer, and a Sigmoid function. The input feature map is connected to the 3*3 convolutional layer, and the 3*3 convolutional layer, the activation layer, and the 1*1 convolutional layer are connected in sequence; the output feature map of the 1*1 convolutional layer is respectively connected to the average pooling layer and the maximum pooling layer, the output of the average pooling layer obtains a first one-dimensional vector, and the output of the maximum pooling layer obtains a second one-dimensional vector. The first one-dimensional vector and the second one-dimensional vector are superimposed and normalized into a 0-1 vector by a Sigmoid function. The normalized vector is multiplied by the output feature map of the 1*1 convolutional layer and then added to the input feature map to obtain the output feature of the residual block.

[0018] Furthermore, the eight residual blocks are connected in a densely connected manner, that is, the outputs of all previous residual blocks are used as inputs of the next residual block.

[0019] Further, the feature extraction part also includes a spatial pyramid pooling module, which is connected between the residual block and the fourth deconvolution layer; the spatial pyramid pooling module includes a first 1*1 convolution layer, a 3*3 dilated convolution layer with an expansion rate of 6, a 3*3 dilated convolution layer with an expansion rate of 12, a 3*3 dilated convolution layer with an expansion rate of 18, an average pooling layer, and a second 1*1 convolution. The first 1*1 convolution layer, the 3*3 dilated convolution layer with an expansion rate of 6, the 3*3 dilated convolution layer with an expansion rate of 12, the 3*3 dilated convolution layer with an expansion rate of 18, and the average pooling layer are connected in parallel, and the outputs of the first 1*1 convolution layer, the 3*3 dilated convolution layer with an expansion rate of 6, the 3*3 dilated convolution layer with an expansion rate of 12, the 3*3 dilated convolution layer with an expansion rate of 18, and the average pooling layer are stacked and connected to the second 1*1 convolution layer for channel number adjustment.

[0020] Furthermore, the 3*3 dilated convolution layer with an expansion rate of 6, the 3*3 dilated convolution layer with an expansion rate of 12, and the 3*3 dilated convolution layer with an expansion rate of 18 are connected in a densely connected manner; the output of the 3*3 dilated convolution layer with an expansion rate of 6 and the input feature are stacked as the input of the 3*3 dilated convolution layer with an expansion rate of 12; the output of the 3*3 dilated convolution layer with an expansion rate of 6, the output of the 3*3 dilated convolution layer with an expansion rate of 12, and the input feature are stacked as the input of the 3*3 dilated convolution layer with an expansion rate of 18.

[0021] Furthermore, the loss function L of the generative adversarial network model is oss Including the first discriminator loss function L GAN1 , the second discriminator loss function L GAN2 , cycle loss function L cyc , consistency loss function L ide , whose formulas are shown in (1)-(5):

[0022] L GAN1 =E X～Pdata(X) [logP X ]+E Y～Pdata(Y) [log(1-P X (F(Y)))] (1)

[0023] L GAN2 =E Y～Pdata(Y) [logP Y ]+E X～Pdata(X) [log(1-P Y (G(X)))] (2)

[0024] L cyc =E X～Pdata(X) [F(G(X))-X1]+E Y～Pdata(Y) [G(F(Y))-Y1] (3)

[0025] L ide =E X～Pdata(X) [G(X)-X1]+E Y～Pdata(Y) [F(Y)-Y1] (4)

[0026] L oss =L GAN1 +L GAN2 +λL cyc +βL ide (5)

[0027] Among them, P X , P Y is the probability that the data input to the discriminant model is the real data; X~Pdata(X) represents that X obeys the real data distribution, and Y~Pdata(Y) represents that Y obeys the real data distribution; E(·) is the expectation; ||·||1 represents the L1 norm; λ and β are weighting coefficients.

[0028] (III) Beneficial effects

[0029] Beneficial effects of the present invention: A method for augmenting the data of infrared images of heating fittings based on deep learning, which uses a deep learning method to generate infrared sample images of different styles, not only expands the number of samples, but also can effectively amplify the diversity of sample features. The present invention improves the generator by means of cross-layer connection, dense connection of residual blocks, introduction of attention mechanism in residual blocks, introduction of spatial pyramid pooling module, etc., improves some performance of the generator in the model without significantly increasing the complexity of the model, extracts semantic information of different scales, and obtains better feature extraction capabilities. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0031] Figure 1 This is a flow chart of the sample expansion method of the present invention;

[0032] Figure 2 Generate an adversarial network model structure diagram for the present invention;

[0033] Figure 3 This is a schematic diagram of the generator of the present invention;

[0034] Figure 4 This is a residual block structure diagram of the present invention;

[0035] Figure 5 This is a dense connection structure diagram of the residual block of the present invention;

[0036] Figure 6 This is a module structure diagram of the space pyramid of the present invention;

[0037] Figure 7 It is a structural diagram of the discriminator of the present invention;

[0038] Figure 8 Sample images are generated for the heating images of the suspension clamp and the present invention. DETAILED DESCRIPTION

[0039] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0040] Combination Figure 1 , a method for augmenting hardware heating infrared image data based on deep learning, comprising the following steps:

[0041] S1. Collect infrared images of hardware: Use a drone equipped with infrared cameras to detect the status of power transmission line hardware. The collected inspection images are stored in a memory card or wirelessly transmitted to a local server.

[0042] S2. Establish a hardware heating infrared image dataset: Select the hardware heating infrared image in step S1 and adjust the image to a uniform size of 256×256. The hardware in the dataset may include tension clamps and suspension clamps. Among them, the tension clamps fix the power lines and lightning conductors on the insulator strings of the tension, corner, and terminal towers, and the suspension clamps are used to suspend or support the power lines.

[0043] S3. Build a deep convolutional neural network model: Use python language to build a generative adversarial network model under the pytorch deep learning framework.

[0044] S4. Training a generative adversarial network model: Using transfer learning and the hardware heating infrared image dataset established in step S2, the deep convolutional neural network model built in step S3 is trained and optimized to obtain a generative adversarial network model.

[0045] S5. Generate infrared image samples: Select the heating infrared image of the hardware and input it into the generative adversarial network model trained in step S4 to perform infrared image prediction, thereby completing the infrared image sample expansion.

[0046] Combination Figure 2 The generative adversarial network model in step S3 is a CycleGAN network model, which includes a first generator G, a first discriminator D X , second generator F, second discriminator D Y The hardware heating infrared image data set is divided into X-domain images and Y-domain images. The X-domain images contain real samples X, and the Y-domain images contain real samples Y. The first generator G generates generated samples G(X) and reconstructed samples G(F(Y)), and the second generator F generates generated samples F(Y) and reconstructed samples F(G(X)). The real samples X and generated samples F(Y) are connected to the first generator G to generate generated samples G(X). The generated samples G(X) and the real samples Y are connected to the second discriminator D. Y It is used to judge the authenticity of the generated samples and obtain the loss function L GAN2 The real sample Y and the generated sample G(X) are connected to the second generator F to generate the generated sample F(Y), and the generated sample F(Y) is connected to the real sample X and the first discriminator D X It is used to judge the authenticity of the generated samples and obtain the loss function L GAN1 The reconstructed sample F(G(X)) and the real sample X, the reconstructed sample G(F(Y)) and the real sample Y constitute a cyclic loss function L cyc To further constrain the training process of the generator, a consistency loss function L is introduced ide, the real sample image and the generated sample image are input into the corresponding generator respectively, and the image output by the generator should remain unchanged. The cycle consistency loss is the consistency loss between the reconstructed image and the real image of the generated image obtained by reconstructing the images generated by the first generator G and the second generator F, and the L1 distance is used for measurement. The loss function L of the generative adversarial network model oss This is achieved by weighted summation, and only the values ​​of λ and β need to be adjusted to control the importance of reconstruction consistency. The loss function L of the generative adversarial network model is oss Including the first discriminator loss function L GAN1 , the second discriminator loss function L GAN2 , cycle loss function L cyc , consistency loss function L ide , whose formulas are shown in (1)-(5):

[0047] L GAN1 =E X～Pdata(X) [logP X ]+E Y～Pdata(Y) [log(1-P X (F(Y)))] (1)

[0048] L GAN2 =E Y～Pdata(Y) [logP Y ]+E X～Pdata(X) [log(1-P Y (G(X)))] (2)

[0049] L cyc =E X～Pdata(X) [||F(G(X))-X||1]+E Y～Pdata(Y) [||G(F(Y))-Y||1] (3)

[0050] L ide =E X～Pdata(X) [||G(X)-X||1]+E Y～Pdata(Y) [||F(Y)-Y||1] (4)

[0051] L oss =L GAN1 +L GAN2 +λL cyc +βL ide (5)

[0052] Among them, P X , P Yis the probability that the data input to the discriminant model is the real data; X~Pdata(X) represents that X follows the real data distribution, and Y~Pdata(Y) represents that Y follows the real data distribution; E(·) is the expectation; ||·||1 represents the L1 norm. λ and β are weighting coefficients, λ can be equal to 10, and β can be equal to 5.

[0053] Combination Figure 3 , the first generator G and the second generator F include an encoding part, a feature extraction part, and a decoding part. The encoding part reduces the image size by downsampling, the feature extraction part is used to extract the features of the hardware heating image, and the decoding part restores the image size by interpolation bilinear upsampling. The encoding part, the feature extraction part, and the decoding part are connected in sequence. The encoding part includes four convolutional layers, namely the first convolutional layer, the second convolutional layer, the third convolutional layer, and the fourth convolutional layer. The feature extraction part is formed by connecting 8 residual blocks in sequence. The decoding part includes four deconvolutional layers, namely the first deconvolutional layer, the second deconvolutional layer, the third deconvolutional layer, and the fourth deconvolutional layer. The encoding part and the decoding part are also connected in a cross-layer manner, that is, the output of the first convolutional layer is also connected to the input of the first deconvolutional layer, the output of the second convolutional layer is also connected to the input of the second deconvolutional layer, the output of the third convolutional layer is also connected to the input of the third deconvolutional layer, and the output of the fourth convolutional layer is also connected to the input of the fourth deconvolutional layer.

[0054] Combination Figure 4 , the residual block includes a 3*3 convolution layer, an activation layer, a 1*1 convolution layer, a maximum pooling layer, an average pooling layer, and a Sigmoid function. The input feature map is connected to the 3*3 convolution layer, and the 3*3 convolution layer, the activation layer, and the 1*1 convolution layer are connected in sequence; the output feature map of the 1*1 convolution layer is connected to the average pooling layer and the maximum pooling layer respectively, the output of the average pooling layer obtains a first one-dimensional vector, and the output of the maximum pooling layer obtains a second one-dimensional vector. The first one-dimensional vector and the second one-dimensional vector are superimposed and normalized to a 0-1 vector by a Sigmoid function, and the normalized vector is multiplied by the output feature map of the 1*1 convolution layer and then added to the input feature map to obtain the output feature of the residual block. The present invention adds a lightweight attention mechanism on the basis of the original residual block, and improves the partial performance of the generator in the model without significantly increasing the complexity of the model, so as to obtain better feature extraction capabilities.

[0055] Combination Figure 5 In order to further improve the feature extraction capability of the feature extraction part, the eight residual blocks are connected in a dense connection manner, that is, the outputs of all the previous residual blocks are used as the input of the next residual block. Feature reuse can be effectively achieved through dense connection.

[0056] Combination Figure 6In order to obtain semantic information of different scales, the feature extraction part also includes a spatial pyramid pooling module, which is connected between the residual block and the fourth deconvolution layer. The spatial pyramid pooling module includes a first 1*1 convolution layer, a 3*3 dilated convolution layer with an expansion rate of 6, a 3*3 dilated convolution layer with an expansion rate of 12, a 3*3 dilated convolution layer with an expansion rate of 18, an average pooling layer, and a second 1*1 convolution. The first 1*1 convolution layer, a 3*3 dilated convolution layer with an expansion rate of 6, a 3*3 dilated convolution layer with an expansion rate of 12, a 3*3 dilated convolution layer with an expansion rate of 18, and an average pooling layer are connected in parallel. The outputs of the first 1*1 convolution layer, the 3*3 dilated convolution layer with an expansion rate of 6, the 3*3 dilated convolution layer with an expansion rate of 12, the 3*3 dilated convolution layer with an expansion rate of 18, and the average pooling layer are stacked and connected to the second 1*1 convolution layer to adjust the number of channels.

[0057] In order to achieve feature reuse and further expand the receptive field of the feature layer, the 3*3 dilated convolution layer with a dilation rate of 6, the 3*3 dilated convolution layer with a dilation rate of 12, and the 3*3 dilated convolution layer with a dilation rate of 18 are connected in a densely connected manner; the output of the 3*3 dilated convolution layer with a dilation rate of 6 and the input features are stacked as the input of the 3*3 dilated convolution layer with a dilation rate of 12; the output of the 3*3 dilated convolution layer with a dilation rate of 6, the output of the 3*3 dilated convolution layer with a dilation rate of 12, and the input features are stacked as the input of the 3*3 dilated convolution layer with a dilation rate of 18.

[0058] Combination Figure 7 , the first discriminator and the second discriminator include 5 convolutional layers, which are connected in sequence. The size of the input hardware heating infrared image is 256×256×3. After the 4 groups of convolutional layer operations in the encoding part, the output size is 32×32×256. After feature extraction by the residual block and spatial pyramid pooling module of the feature extraction part, the generated image of 256×256×3 is obtained through the upsampling operation of 4 groups of deconvolution layers in the decoding part and the cross-layer connection operation of the encoding part. The discriminator consists of 5 groups of convolutional layers, with the real image and the generated image as input, the size is fixed to 256×256×3, and after 5 groups of convolutional layer operations, the probability value of the output image being judged as the real image is obtained.

[0059] In order to verify the effectiveness of the generative adversarial network model of the present invention, a network model was built under the pytorch deep learning framework, and the CPU used in the hardware platform was 3.60 GHz. Core TM i9-9900K, with 32GB of total memory, and the GPU is NVIDIA GeForce GTX 3080 with 10G memory. Figure 8(a)(b)(c)(d) show 4 original images of the heating of the suspension clamp and the sample images generated by the method of the present invention, where the left image is the original image and the right image is the generated image. It can be seen from the figure that the generated sample image retains the heating point area of ​​the hardware and can be used for the target detection of hardware heating faults. At the same time, compared with geometric transformations (translation, flipping and elastic deformation, etc.), the generated sample image can effectively amplify the diversity of sample features.

[0060] In summary, the embodiment of the present invention is a method for augmenting the data of infrared images of heating fittings based on deep learning, which includes collecting infrared images of hardware fittings, establishing a data set of infrared images of heating fittings, building a deep convolutional neural network model, training a generative adversarial network model, and generating infrared image samples. The present invention uses a deep learning method to generate infrared sample images of different styles, which not only expands the number of samples, but also can effectively amplify the diversity of sample features. The present invention improves the generator by means of cross-layer connections, dense connections of residual blocks, introduction of an attention mechanism in residual blocks, introduction of a spatial pyramid pooling module, etc., thereby improving some performance of the generator in the model without significantly increasing the complexity of the model, extracting semantic information of different scales, and obtaining better feature extraction capabilities.

[0061] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for augmenting hardware heating infrared image data based on deep learning, characterized in that: The following steps are involved: S1. Collect infrared images of hardware: Use a drone equipped with infrared cameras to detect the status of power transmission line hardware. The collected inspection images are stored in a memory card or wirelessly transmitted to a local server. S2. Establishing a hardware heating infrared image dataset: Select the hardware heating infrared image in step S1 and adjust the image to a uniform size of 256×256; S3. Build a deep convolutional neural network model: Use Python language to build a generative adversarial network model under the pytorch deep learning framework; S4, training a generative adversarial network model: using transfer learning and the hardware heating infrared image dataset established in step S2 to train and optimize the deep convolutional neural network model built in step S3, to obtain a generative adversarial network model; S5. Generate infrared image samples: Select the heating infrared image of the hardware and input it into the generative adversarial network model trained in step S4 to perform infrared image prediction, thereby completing the infrared image sample expansion.

2. The method for augmenting the heating infrared image data of hardware based on deep learning as claimed in claim 1, characterized in that: The hardware in the data set of step S2 includes tension clamps and suspension clamps.

3. The method for augmenting the data of heating infrared images of hardware based on deep learning as claimed in claim 1, characterized in that: The generative adversarial network model in step S3 is a CycleGAN network model, which includes a first generator G, a first discriminator D X , second generator F, second discriminator D Y The hardware heating infrared image data set is divided into X-domain images and Y-domain images, the X-domain images contain real samples X, and the Y-domain images contain real samples Y; the first generator G generates generated samples G(X) and reconstructed samples G(F(Y)), and the second generator F generates generated samples F(Y) and reconstructed samples F(G(X)); the real samples X and generated samples F(Y) are connected to the first generator G to generate generated samples G(X), and the generated samples G(X) and the real samples Y are connected to the second discriminator D Y It is used to judge the authenticity of the generated samples and obtain the loss function L GAN2 The real sample Y and the generated sample G(X) are connected to the second generator F to generate the generated sample F(Y), and the generated sample F(Y) is connected to the real sample X and the first discriminator D X It is used to judge the authenticity of the generated samples and obtain the loss function L GAN1 The reconstructed sample F(G(X)) and the real sample X, the reconstructed sample G(F(Y)) and the real sample Y constitute a cyclic loss function L cyc .

4. A method for augmenting hardware heating infrared image data based on deep learning as claimed in claim 3, characterized in that: The first generator G and the second generator F include an encoding part, a feature extraction part, and a decoding part. The encoding part reduces the image size by downsampling, the feature extraction part is used to extract the heating image features of the hardware, and the decoding part restores the image size by interpolation bilinear upsampling. The encoding part, the feature extraction part, and the decoding part are connected in sequence; the encoding part includes four convolutional layers, namely, a first convolutional layer, a second convolutional layer, a third convolutional layer, and a fourth convolutional layer. The feature extraction part is composed of 8 residual blocks connected in sequence, and the decoding part includes four deconvolutional layers, namely, a first deconvolutional layer, a second deconvolutional layer, a third deconvolutional layer, and a fourth deconvolutional layer; the encoding part and the decoding part are also connected in a cross-layer manner, that is, the output of the first convolutional layer is also connected to the input of the first deconvolutional layer, the output of the second convolutional layer is also connected to the input of the second deconvolutional layer, the output of the third convolutional layer is also connected to the input of the third deconvolutional layer, and the output of the fourth convolutional layer is also connected to the input of the fourth deconvolutional layer.

5. The method for augmenting the data of heating infrared images of hardware based on deep learning as claimed in claim 4, characterized in that: The residual block includes a 3*3 convolution layer, an activation layer, a 1*1 convolution layer, a maximum pooling layer, an average pooling layer, and a Sigmoid function. The input feature map is connected to the 3*3 convolution layer, and the 3*3 convolution layer, the activation layer, and the 1*1 convolution layer are connected in sequence; the output feature map of the 1*1 convolution layer is respectively connected to the average pooling layer and the maximum pooling layer, the output of the average pooling layer obtains a first one-dimensional vector, and the output of the maximum pooling layer obtains a second one-dimensional vector. The first one-dimensional vector and the second one-dimensional vector are superimposed and normalized into a 0-1 vector by a Sigmoid function. The normalized vector is multiplied by the output feature map of the 1*1 convolution layer and then added to the input feature map to obtain the output feature of the residual block.

6. A method for augmenting hardware heating infrared image data based on deep learning as claimed in claim 4, characterized in that: The eight residual blocks are connected in a densely connected manner, that is, the outputs of all previous residual blocks serve as the inputs of the next residual block.

7. The method for augmenting hardware heating infrared image data based on deep learning as claimed in claim 4, characterized in that: The feature extraction part also includes a spatial pyramid pooling module, which is connected between the residual block and the fourth deconvolution layer; the spatial pyramid pooling module includes a first 1*1 convolution layer, a 3*3 dilated convolution layer with an expansion rate of 6, a 3*3 dilated convolution layer with an expansion rate of 12, a 3*3 dilated convolution layer with an expansion rate of 18, an average pooling layer, and a second 1*1 convolution. The first 1*1 convolution layer, the 3*3 dilated convolution layer with an expansion rate of 6, the 3*3 dilated convolution layer with an expansion rate of 12, the 3*3 dilated convolution layer with an expansion rate of 18, and the average pooling layer are connected in parallel, and the outputs of the first 1*1 convolution layer, the 3*3 dilated convolution layer with an expansion rate of 6, the 3*3 dilated convolution layer with an expansion rate of 12, the 3*3 dilated convolution layer with an expansion rate of 18, and the average pooling layer are stacked and connected to the second 1*1 convolution layer to adjust the number of channels.

8. The method for augmenting the data of heating infrared images of hardware based on deep learning as claimed in claim 7, characterized in that: The 3*3 dilated convolution layer with an expansion rate of 6, the 3*3 dilated convolution layer with an expansion rate of 12, and the 3*3 dilated convolution layer with an expansion rate of 18 are connected in a densely connected manner; the output of the 3*3 dilated convolution layer with an expansion rate of 6 and the input feature are stacked as the input of the 3*3 dilated convolution layer with an expansion rate of 12; the output of the 3*3 dilated convolution layer with an expansion rate of 6, the output of the 3*3 dilated convolution layer with an expansion rate of 12, and the input feature are stacked as the input of the 3*3 dilated convolution layer with an expansion rate of 18.

9. The method for augmenting hardware heating infrared image data based on deep learning as claimed in claim 3, characterized in that: The loss function L of the generative adversarial network model is oss Including the first discriminator loss function L GAN1 , the second discriminator loss function L GAN2 , the cycle loss function L cyc , consistency loss function L ide , whose formulas are shown in (1)-(5): L GAN1 =E X～Pdata(X) [logP X ]+E Y～Pdata(Y) [log(1-P X (F(Y)))] (1) THE GAN2 =And Y～Pdata(Y) [logP Y ]+E X～Pdata(X) [log(1-P Y (G(X)))] (2) L cyc =E X～Pdata(X) [||F(G(X))-X||1]+E Y～Pdata(Y) [||G(F(Y))-Y||1] (3) L ide =E X～Pdata(X) [||G(X)-X||1]+E Y～Pdata(Y) [||F(Y)-Y||1] (4) L oss =L GAN1 +L GAN2 +λL cyc +βL ide (5) Among them, P X , P Y is the probability that the data input to the discriminant model is the real data; X~Pdata(X) represents that X obeys the real data distribution, and Y~Pdata(Y) represents that Y obeys the real data distribution; E(·) is the expectation; ||·||1 represents the L1 norm; λ and β are weighting coefficients.

Images