Infrared image temperature matrix mapping method and system based on nonlinear regression network

Abstract

The application discloses an infrared image temperature matrix mapping method and system based on a nonlinear regression network. The method comprises inputting an infrared pseudo-color image into a trained nonlinear regression network to obtain a temperature matrix. The nonlinear regression network comprises a preprocessing module, a convolution module, a full connection module and a post-processing module. The processing of the input infrared pseudo-color image comprises: reading the infrared pseudo-color image by the preprocessing module and performing preprocessing including normalization; performing channel dimension reduction to a grayscale image for the preprocessed infrared pseudo-color image by the convolution module; performing flattening and batch normalization on the grayscale image, and then calculating a preliminary temperature matrix by the full connection module; and performing post-processing including inverse standard normalization on the preliminary temperature matrix by the post-processing module to obtain the temperature matrix. The application aims to realize reverse interpretation from an infrared pseudo-color image to a temperature matrix, and realize general decoupling of infrared camera software and hardware.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of infrared image processing, and particularly relates to an infrared image temperature matrix mapping method and system based on a nonlinear regression network. BACKGROUND

[0002] The power transformation equipment is long-term operated in the open environment, is affected by the wind and the sun, is easy to have a fault and then causes the power grid shutdown event, and the common power transformation equipment fault early performance is abnormal heating, therefore, the monitoring and the monitoring of the operation temperature of the power transformation equipment are important contents of the power transformation operation and maintenance work. The infrared thermal imaging temperature measurement has many advantages such as non-contact, easy implementation, fast response, strong intuition, etc., and is widely applied in the transformer substation. In the early stage, the operation and maintenance personnel take close-up pictures of the power transformation equipment by the handheld infrared imager, then import the infrared picture into the special software, and analyze the operation condition of the equipment combined with the analyzed temperature matrix. In this way, the work intensity of the operation and maintenance personnel is large, and time and effort are consumed. In recent years, with the emergence of various digital technologies, the infrared camera gradually replaces the handheld infrared imager and becomes the main means of the operation state monitoring of the power transformation equipment. The infrared camera transmits the video information and the temperature information to the middle station monitoring system through the network, and the operation and maintenance personnel can master the operation condition of the on-site equipment in the centralized control room, which greatly improves the work efficiency. At present, the infrared camera brands widely applied in the transformer substation include Hikvision, Dahua, and Dali, etc., but the temperature data acquisition of each brand depends on the software of each brand, and the software development kit must be configured at the same time as the hardware product of each brand is used. The software and hardware binding mode leads to high price of the infrared camera on the one hand, and also aggravates the bloated monitoring system, which is not conducive to the subsequent iteration and upgrading. Therefore, a universal method capable of realizing the software and hardware decoupling of the infrared camera is needed to acquire the temperature data of different brand infrared cameras.

[0003] From the technical principle of the infrared thermal imaging, the sensor converts the radiation intensity into the temperature information after receiving the infrared radiation energy of the measured object, but in order to more intuitively display, the temperature information is mapped into the gray scale image, and the RGB color image is further obtained through the pseudo-color enhancement. From the temperature matrix to the pseudo-color image, two nonlinear mapping processes are experienced, and conversely, the pseudo-color image can be restored to the temperature matrix through two inverse mappings. The artificial neural network has strong fitting capability, and the application of this characteristic to realize the inversion from the pseudo-color image to the temperature matrix is a feasible path. SUMMARY

[0004] The technical problem to be solved by the present application: in view of the above problems of the prior art, a kind of infrared image temperature matrix mapping method and system based on nonlinear regression network are provided, the present application aims to realize the reverse interpretation from infrared pseudo-color image to temperature matrix, and realize the universal decoupling of the software and hardware of infrared camera.

[0005] To solve the above technical problems, the technical scheme adopted by the present application is:

[0006] An infrared image temperature matrix mapping method based on a nonlinear regression network, comprising inputting an infrared pseudo-color image into a trained nonlinear regression network to obtain a temperature matrix, the nonlinear regression network comprising a preprocessing module, a convolution module, a full connection module, and a post-processing module, and the processing of the input infrared pseudo-color image comprising:

[0007] 1) reading the infrared pseudo-color image using the preprocessing module and performing preprocessing including normalization;

[0008] 2) using the convolution module to reduce the channel dimension of the preprocessed infrared pseudo-color image to a grayscale image;

[0009] 3) using the full connection module to calculate a preliminary temperature matrix after the grayscale image is flattened and batch normalized;

[0010] 4) using the post-processing module to perform post-processing including inverse standard normalization on the preliminary temperature matrix to obtain the temperature matrix.

[0011] Optionally, step 1) comprises:

[0012] 1.1) reading the infrared pseudo-color image to obtain a three-dimensional array M1 with dimensions [3, height, width], where 3 is the color channel of the image, height is the height of the image, and width is the width of the image;

[0013] 1.2) performing standard normalization on the three-dimensional array M1 to obtain a three-dimensional array M2 with dimensions [3, height, width], the relevant parameters of standard normalization including mean and variance;

[0014] 1.3) calculating the remainders r h , r w of the height of the image height and the width of the image width, respectively, with respect to a given parameter a; if the remainders r h are not equal to 0, then a-r h pixels are symmetrically padded in the height direction, and the padding values are M2[:,0,:] and M2[:,-1,:], respectively, to obtain a three-dimensional array M3 with dimensions [3,H,width], where the height H satisfies H=height+a-r h ; if the remainders r w are not equal to 0, then a-r wM3[:,:,0] and M3[:,0,-1] respectively, to obtain a three-dimensional array M4 with the padded dimension of [3, H, W], where the width W satisfies W = width + a - r w ; cropping the three-dimensional array M4 with the size of a*a, and splicing on the new dimension to obtain a four-dimensional array M5 with the dimension of [b, 3, a, a] as the output feature of the pre-processing, where the variable b = H / a*W / a.

[0015] Optionally, in step 2), when the channel dimension of the pre-processed infrared pseudo-color image is reduced to a gray image by using a convolution module, the convolution module comprises one or more convolution layers, each of which is a convolution kernel with a size of 1*1 followed by a Relu activation function, the input channel number of the first convolution layer of the convolution module is 3, the output channel number of the last convolution layer of the convolution module is 1, and the four-dimensional array M5 is processed by the convolution module to obtain a four-dimensional array M6 with the dimension of [b, 1, a, a] as the output gray image after channel dimension reduction.

[0016] Optionally, the flattening and batch normalization processing of the gray image in step 3) comprises:

[0017] 3.1) flattening the four-dimensional array M6 with the dimension of [b, 1, a, a] as the output gray image after channel dimension reduction, retaining the dimension 0 to obtain a two-dimensional array M7 with the dimension of [b, d], where the variable d = 1*a*a;

[0018] 3.2) batch standard normalizing the two-dimensional array M7 to obtain a two-dimensional array M8 with the dimension of [b, d].

[0019] Optionally, the function expression of batch standard normalizing the two-dimensional array M7 in step 3.2) is:

[0020]

[0021] In the above formula, is the lth element of the bs batch of the two-dimensional array M8, is the lth element of the bs batch of the two-dimensional array M7, γ is a scale factor, μ is the mean of the two-dimensional array M7, σ is the standard deviation of the two-dimensional array M7, ε is a parameter close to 0, β is a translation factor, and the mean, variance, scale factor and translation factor are learning parameters of the nonlinear regression network.

[0022] Optionally, when the preliminary temperature matrix is calculated in step 3) by using the full connection module, the full connection module comprises one or more full connection layers, the input dimension of the first full connection layer of the full connection module is d, the output dimension of the last full connection layer of the full connection module is d, and a two-dimensional array M9 with a dimension of [b, d] is obtained after processing by the full connection module to serve as the preliminary temperature matrix.

[0023] Optionally, step 4) comprises:

[0024] 4.1) performing dimension transformation on the two-dimensional array M9 with a dimension of [b, d] representing the preliminary temperature matrix to obtain a three-dimensional array M 10 ;

[0025] 4.2) performing inverse splicing on the three-dimensional array M 10 to obtain a two-dimensional array M 11 ;

[0026] 4.3) removing the padding around the two-dimensional array M 11 to obtain a two-dimensional array M 12 ;

[0027] 4.4) performing inverse standard normalization on the two-dimensional array M 12 to obtain a final temperature matrix M 13 with a dimension of [height, width], and the related parameters of the inverse normalization include mean and variance.

[0028] Optionally, before step 1), a nonlinear regression network is constructed and trained:

[0029] S1, preparing infrared pseudo-color image samples and temperature matrices serving as labels thereof, and constructing a training data set;

[0030] S2, constructing a nonlinear regression network and initializing network weight parameters of the nonlinear regression network;

[0031] S3, inputting the infrared pseudo-color images in the training data set into the nonlinear regression network for forward calculation and processing, calculating the regression loss between the temperature matrix output by the nonlinear regression network and the temperature matrix serving as the label, calculating the gradient of the loss function with respect to each network layer neuron weight according to the regression loss, and updating the network weight parameters of the nonlinear regression network, and repeating the above steps until a specified number of training is completed or the regression loss is less than a set value.

[0032] Further, the present application also provides an infrared image temperature matrix mapping system based on a nonlinear regression network, comprising a microprocessor and a memory connected to each other, the microprocessor being programmed or configured to perform the steps of the infrared image temperature matrix mapping method based on a nonlinear regression network.

[0033] Further, the present application also provides a computer readable storage medium having stored therein a computer program programmed or configured to perform the infrared image temperature matrix mapping method based on a nonlinear regression network.

[0034] Compared with the prior art, the present application has the following advantages:

[0035] 1. The present application uses a nonlinear regression network to invert the temperature matrix of an infrared image, which is consistent with the principle of infrared thermal imaging, can realize the reverse interpretation from an infrared pseudo-color image to a temperature matrix, realize the general decoupling of the hardware and software of an infrared camera, and has a simple process, strong interpretability, high conversion accuracy, and can meet the quantitative analysis and diagnosis requirements of abnormal heating defects of power transformation equipment; moreover, the method of the present application is simple, does not depend on specific software, and has strong universality, which helps to reduce the labor intensity of power transformation station operation and maintenance personnel, improve the operation and maintenance efficiency and the automation and intelligent level of power transformation equipment.

[0036] 2. The method of the present application for inverting the temperature matrix of an infrared image using a nonlinear regression network is a learning-based method, the model capability of which depends on the training data and is not dependent on specific brands of hardware or software, and has high flexibility; at the same time, it is effective for infrared picture data or video stream data, can be conveniently deployed on a cloud server or a terminal device, is not limited by an operating system, and has wide applicability. BRIEF DESCRIPTION OF DRAWINGS

[0037] Figure 1 is a flowchart of the method of an embodiment of the present application.

[0038] Figure 2 is a workflow diagram of the nonlinear regression network in an embodiment of the present application.

[0039] Figure 3 is the deviation between the temperature matrix parsed by the method of an embodiment of the present application and the method of DJI TSDK. DETAILED DESCRIPTION

[0040] As shown in Figure 1 , the infrared image temperature matrix mapping method based on a nonlinear regression network of the present embodiment comprises inputting an infrared pseudo-color image into a trained nonlinear regression network to obtain a temperature matrix, the nonlinear regression network comprising a preprocessing module, a convolution module, a full connection module and a post-processing module.

[0041] AsFigure 2 As shown, and the processing of the input infrared pseudo-color image by the nonlinear regression network includes:

[0042] 1) reading the infrared pseudo-color image by using the preprocessing module and performing preprocessing including normalization;

[0043] 2) performing channel dimension reduction to a grayscale image for the preprocessed infrared pseudo-color image by using the convolution module;

[0044] 3) performing flattening and batch normalization on the grayscale image, and then calculating a preliminary temperature matrix by using the full connection module;

[0045] 4) performing post-processing including inverse standard normalization on the preliminary temperature matrix by using the post-processing module to obtain the temperature matrix.

[0046] In this embodiment, step 1) includes:

[0047] 1.1) reading the infrared pseudo-color image to obtain a three-dimensional array M1 with dimensions [3, height, width], where 3 is the color channel of the image, height is the height of the image, and width is the width of the image; in this embodiment, the opencv is used to read the infrared pseudo-color image to obtain the three-dimensional array M1 with dimensions [3, 480, 640];

[0048] 1.2) performing standard normalization on the three-dimensional array M1 to obtain a three-dimensional array M2 with dimensions [3, height, width], and the related parameters of the standard normalization include the mean and the variance; in this embodiment, the three-dimensional array M1 is standardized to obtain the three-dimensional array M2 with dimensions [3, 480, 640], and the mean and the standard deviation of the standard normalization are obtained from the training set image data; the function expression of the standard normalization is:

[0049]

[0050] In the above formula, represents the element of the i-th channel, the j-th row, and the k-th column of the two-dimensional array M2, represents the element of the i-th channel, the j-th row, and the k-th column of the two-dimensional array M1, μ i is the mean value of the training set image data by channel, σ i is the standard deviation of the training set image data by channel, where i = 0, 1, 2, i = 0, 1, 2,..., 479, and k = 0, 1, 2,..., 639;

[0051] 1.3) respectively calculating the remainders r h of the height of the image height with respect to the given parameter a, wIf the remainder r h If not equal to 0, then fill ar symmetrically in the height direction. h Given 12 pixels, filled with values ​​M2[:,0,:] and M2[:,-1,:] respectively, we obtain a 3D array M3 with dimensions [3,H,width], where the height H satisfies H=height+ar h If the remainder r w If not equal to 0, then fill 'ar' symmetrically in the width direction. w With 1 pixel, and fill values ​​M3[:,:,0] and M3[:,0,-1] respectively, we obtain a three-dimensional array M4 with dimensions [3,H,W] after filling, where the width W satisfies W=width+ar w The three-dimensional array M4 is clipped to size a*a and then concatenated in the new dimension to obtain a four-dimensional array M5 with dimensions [b,3,a,a] as the output feature of the preprocessing, where the variable b = H / a*W / a. The parameter 'a' can be set according to actual needs. In actual business, the size of infrared images is mostly 512*512, 640*480, and 640*512. Parameter 'a' of 128 is divisible in most cases, and even if it is not divisible, the remainder is not large. Therefore, in this embodiment, the parameter 'a' is set to 128. The remainders of 480 and 640 divided by 128 are 96 and 0, respectively. 32 pixels are symmetrically filled in the height direction, with fill values ​​of M2[:,0,:] and M2[:,-1,:], respectively, to obtain a three-dimensional array M3 with dimensions [3,512,640]. M3 is cropped to a size of 128*128 and spliced ​​in the new dimension direction to obtain a four-dimensional array M5 with dimensions [20,3,128,128].

[0052] In step 2) of this embodiment, when using a convolutional module to reduce the channel dimension of the preprocessed infrared pseudo-color image to grayscale, the convolutional module contains one or more convolutional layers. Each convolutional layer consists of a 1×1 kernel followed by a ReLU activation function. The first convolutional layer has 3 input channels, and the last convolutional layer has 1 output channel. After processing by the convolutional module, the four-dimensional array M5 yields a four-dimensional array M6 with dimensions [b,1,a,a], which serves as the grayscale image output after channel dimension reduction. As an optional implementation, the convolutional module in this embodiment contains two 1×1 kernel convolutional layers: convolutional layer #1 and convolutional layer #2. The input and output channels of convolutional layer #1 are 3 and 16, respectively, and the input and output channels of convolutional layer #2 are 16 and 1, respectively. The calculation formula for the convolution operation is:

[0053]

[0054]

[0055] In the formula, m' bs,c,i,j denotes the element indexed as [bs, c, i, j] of the four-dimensional array M5 after convolution by convolution layer #1, Relu is a Relu activation function, and m 5 bs,k,i,j denotes the element indexed as [bs, k, i, j] of the four-dimensional array M5, CONV1 k,c denotes the convolution kernel of the cth channel of convolution layer #1, b CONV1,c is the bias term of convolution layer #1, and Relu is an activation function; m 6 bs,1,i,j denotes the element indexed as [bs, 1, i, j] of the four-dimensional array M6 after convolution by convolution layer #2. CONV2 c denotes the convolution kernel of the 1st channel of convolution layer #2, b CONV2 is the bias term of convolution layer #2, bs = 0, 1, 2, …, 19, c = 0, 1, 2, …, 15, i = 0, 1, 2, …, 127, and j = 0, 1, 2, …, 127.

[0056] The flattening and batch normalization processing of the gray-scale image in step 3) of this embodiment includes:

[0057] 3.1) Flattening the four-dimensional array M6 of the gray-scale image output after channel dimension reduction with the dimension [b, 1, a, a] and keeping the 0th dimension, to obtain a two-dimensional array M7 with the dimension [b, d], where the variable d = 1*a*a.

[0058] 3.2) Batch standard normalization of the two-dimensional array M7 to obtain a two-dimensional array M8 with the dimension [b, d].

[0059] In this embodiment, the four-dimensional array M6 is flattened after keeping the 0th dimension to obtain a two-dimensional array M7 with the dimension [20, 16384]; and the two-dimensional array M7 is batch standard normalized to obtain a two-dimensional array M8 with the dimension [20, 16384].

[0060] In this embodiment, the function expression for batch standard normalization of the two-dimensional array M7 in step 3.2) is:

[0061]

[0062] In the formula, is the lth element (d = 16384, so l = 0, 1, 2, …, 16383) of the bth batch of the two-dimensional array M8, wherein M7 is the bs-th batch of the l-th element of the two-dimensional array M7, γ is a scale factor, μ is the mean of the two-dimensional array M7, σ is the standard deviation of the two-dimensional array M7, ε is a parameter close to 0, β is a translation factor, and the mean, the variance, the scale factor, and the translation factor are learning parameters of the nonlinear regression network, and their specific values are learned in the model training.

[0063] In step 3) of the embodiment, the preliminary temperature matrix is calculated by using a full connection module, wherein the full connection module comprises one or more full connection layers, the input dimension of the first full connection layer of the full connection module is d, and the output dimension of the last full connection layer of the full connection module is d. After the full connection module processing, a two-dimensional array M9 with a dimension of [b, d] is obtained as the preliminary temperature matrix. Specifically, in the embodiment, the full connection module comprises two full connection layers: full connection layer #1 and full connection layer #2, wherein the input and output dimensions of the full connection layer #1 are 16384 and 16 respectively, and the input and output dimensions of the full connection layer #2 are 16 and 16384 respectively. The function expression of the full connection operation is:

[0064]

[0065]

[0066] In the above formula, m' bs,k represents the bs-th batch, k-th element after the full connection layer #1 processing; Relu is the Relu activation function, w l is the l-th weight coefficient of the full connection layer #1, M8 is the bs-th batch, l-th element of the two-dimensional array M8, b FC1 is the bias term of the full connection layer #1, m 9 bs,l represents the bs-th batch, l-th element of the two-dimensional array M9 after the full connection layer #2 processing, w k is the k-th weight coefficient of the full connection layer #2, b FC2 is the bias term of the full connection layer #2, and k = 0, 1, 2, …, 15, l = 0, 1, 2, …, 16383.

[0067] Step 4) of the embodiment comprises:

[0068] 4.1) performing dimension transformation on the two-dimensional array M9 with a dimension of [b, d] representing the preliminary temperature matrix to obtain a three-dimensional array M 10 with a dimension of [b, a, a];

[0069] 4.2) performing inverse splicing on the three-dimensional array M 10 to obtain a two-dimensional array M 11 with a dimension of [H, W];

[0070] 4.3) removing the padding of the two-dimensional array M 11 Four weeks of padding, resulting in a two-dimensional array M of dimension [height, width] 12 ;

[0071] 4.4) inverse standard normalization of the two-dimensional array M 12 , resulting in the final temperature matrix M of dimension [height, width] 13 , the inverse normalization-related parameters including mean and variance.

[0072] In steps 4.1) to 4.4) of this embodiment, the dimension transformation is performed on the forward calculation result (preliminary temperature matrix) - a two-dimensional array M9 of dimension [20, 16384], resulting in a three-dimensional array M of dimension [20, 128, 128] 10 ; reverse splicing is performed on the three-dimensional array M 10 , resulting in a two-dimensional array M of dimension [512, 640] 11 ; removing the 32 padding rows in the height direction of the two-dimensional array M 11 , resulting in a two-dimensional array M of dimension [480, 640] 12 ; inverse standard normalization is performed on the two-dimensional array M 12 , resulting in the final temperature matrix M of dimension [480, 640] 13 . The function expression for inverse standard normalization of the two-dimensional array M 12 in step 3.4) is:

[0073]

[0074] In the above formula, denotes the element in the i-th row and j-th column of the two-dimensional array M 13 , and σ' is the standard deviation of the training set temperature data, denotes the element in the i-th row and j-th column of the two-dimensional array M 12 , and μ' is the mean of the training set temperature data.

[0075] Before step 1) of this embodiment, it also includes constructing and training a nonlinear regression network:

[0076] S1, prepare infrared pseudo-color image samples and their temperature matrices as labels, and construct a training data set;

[0077] S2, construct a nonlinear regression network and initialize the network weight parameters of the nonlinear regression network;

[0078] S3, inputting the infrared pseudo-color image in the training data set into the nonlinear regression network for forward calculation and processing, calculating the regression loss between the temperature matrix output by the nonlinear regression network and the temperature matrix as the label, calculating the gradient of the loss function with respect to the neural weight of each network layer according to the regression loss, and updating the network weight parameter of the nonlinear regression network, and repeating the above steps until a specified number of training is completed or the regression loss is less than a set value.

[0079] Step S1 in this embodiment includes: collecting infrared images taken by infrared cameras or cameras of various brands; applying temperature analysis tools of various brands to interpret the temperature matrix of the infrared image; saving the temperature matrix as a csv file, one csv file corresponding to one infrared image, and the file name being the same as the infrared image name.

[0080] Step S2 in this embodiment includes constructing a nonlinear regression network, which includes: S21) obtaining the maximum high and wide values of the data set image; S22) respectively calculating the mean and standard deviation of the three-dimensional matrix and the temperature matrix of the data set image; S23) constructing a custom preprocessing module based on the transforms class of pytorch, including three processes of standard normalization, padding, and cropping and splicing of the three-dimensional matrix and the temperature matrix of the picture; S24) constructing a custom data set based on the Dataset class of pytorch; S25) constructing a convolution module based on the Conv2d function and the ReLU function of pytorch, the input and output channel numbers of the convolution layer #1 being 3 and 16 respectively, and the input and output channel numbers of the convolution layer #2 being 16 and 1 respectively; S26) implementing the flattening and batch standardization operation based on the view function and the BatchNorm2d function of pytorch; S27) constructing a fully connected module based on the Linear function and the ReLU function of pytorch, the input and output dimensions of the fully connected layer #1 being 16384 and 16 respectively, and the input and output dimensions of the fully connected layer #2 being 16 and 16384 respectively; S28) constructing a custom post-processing module based on the transforms class of pytorch, including a dimension transformation process. In step S2 of this embodiment, when initializing the network weight parameters of the nonlinear regression network, the convolution layer adopts the Xavier initialization method; the batch normalization layer adopts the equal value initialization, and the padding value is 1; and for the fully connected layer, the normal initialization method is adopted.

[0081] Step S3 in this embodiment includes: constructing a data loading method based on the Dataloader class of pytorch; loading the training data in batches, and executing the forward calculation process of S25 to S28. And the regression loss between the temperature matrix output by the nonlinear regression network and the temperature matrix as the label is calculated by using the root mean square error to calculate the regression loss, and the calculation formula of the root mean square error of one Dataloader loading batch is:

[0082]

[0083] In the above formula, RMSE represents the root mean square error of a Dataloader loading batch, B is the batch_size (batch) size, n is the number of elements in a batch, y i is the forward calculation result of the bth batch, the ith element, is the temperature label value corresponding to the bth batch, the ith element. In addition, the network weight parameters of the nonlinear regression network are updated in step S3 using the random gradient descent method to update the network weight. The end condition of completing a specified number of training or the regression loss being less than a set value is that the difference between the regression losses of the adjacent two rounds is less than 0.0001. After the training is completed, the trained nonlinear regression network is used to perform temperature inversion on the infrared image (640*512) taken by the DJI H20T (DJI_H20T) camera, and the obtained temperature matrix is compared with the temperature matrix parsed by the method provided by the DJI TSDK to obtain the results as shown in Figure 3 Figure 3 , the average temperature deviation between the method of the present embodiment and the temperature matrix parsed by the method provided by the DJI TSDK is 0.12℃, and the maximum single-point temperature deviation is 0.5℃. It can be seen that the nonlinear regression network can realize accurate reverse interpretation of the infrared pseudo-color image to the temperature matrix.

[0084] ​In summary, the current infrared imager or infrared camera is used for the maintenance of the power transformation equipment, which is limited by the hardware and software binding mode, high cost, and the system is bulky, and the subsequent upgrade is difficult. According to the principle of infrared thermal imaging, the infrared image temperature matrix mapping method based on nonlinear regression network of the embodiment includes reading the three-dimensional array of the infrared pseudo-color image, pre-processing the three-dimensional array; adjusting the color channel of the pseudo-color image through the convolution module, and transforming the RGB image into a gray image; the gray image is flattened and batch normalized; the processed array is input into the full connection module, and the preliminary result is obtained through forward calculation; the preliminary calculation result is post-processed to obtain the final temperature matrix. The method of the embodiment uses the nonlinear regression idea to inverse the temperature matrix of the infrared image, which is consistent with the principle of infrared thermal imaging, the process is simple, the interpretability is strong, and the conversion accuracy is high, which can meet the quantitative analysis and diagnosis demand of the abnormal heating defect of the power transformation equipment; the method of the embodiment is a learning-based method, the model capability depends on the training data and is not dependent on specific brand hardware or software, and has high flexibility; at the same time, the infrared picture data or video stream data is effective, and can be conveniently deployed on the cloud server or terminal equipment, and is not limited by the operating system, and has wide applicability. The infrared image temperature matrix mapping method based on nonlinear regression network of the embodiment inverses two nonlinear processes of three-dimensional pseudo-color enhancement and temperature-gray mapping through the convolution module and the full connection module, realizes the reverse interpretation from the infrared pseudo-color image to the temperature matrix, and has the advantages of simple method, no dependence on specific software, and strong universality.

[0085] In addition, the embodiment also provides an infrared image temperature matrix mapping system based on a nonlinear regression network, comprising a microprocessor and a memory connected to each other, the microprocessor is programmed or configured to execute the steps of the infrared image temperature matrix mapping method based on the nonlinear regression network. In addition, the embodiment also provides a computer readable storage medium, and the computer readable storage medium stores a computer program programmed or configured to execute the infrared image temperature matrix mapping method based on the nonlinear regression network.

[0086] Those skilled in the art will appreciate that embodiments of the application can be readily used as software, hardware, or a combination of software and hardware. In a software embodiment, various software modules in accordance with embodiments of the application are stored in a memory such as a computer program product (e.g., a disk storage medium) and executed by a computer processor. As such, various computer program products can be supplied with a computer program product embodying computer programs of software or programs of code designed to carry out the methods provided by embodiments of the application. In a hardware embodiment, embodiments of the application can be provided as one or more application specific circuits (ASICs) or field programmable gate arrays (FPGAs). Figure 1 one or more functions specified in the flow or flows and / or blocks Figure 1 one or more functions specified in the flow or flows and / or blocks Figure 1 one or more functions specified in the flow or flows and / or blocks Figure 1 one or more functions specified in the flow or flows and / or blocks Figure 1 one or more functions specified in the flow or flows and / or blocks Figure 1 one or more functions specified in the flow or flows and / or blocks

[0087] The above description is only preferred embodiments of the application. The protection scope of the application is not limited to the above-mentioned embodiments. Any technical scheme falling within the concept of the application shall be considered as falling within the protection scope of the application. It should be noted that any improvement and modification made by those skilled in the art without departing from the principle of the application shall also be considered as falling within the protection scope of the application.

Claims

1. A method for mapping the temperature matrix of infrared images based on a nonlinear regression network, characterized in that, The process includes inputting an infrared pseudo-color image into a trained nonlinear regression network to obtain a temperature matrix. The nonlinear regression network includes a preprocessing module, a convolutional module, a fully connected module, and a post-processing module. The processing of the input infrared pseudo-color image includes: 1) Use the preprocessing module to read the infrared pseudo-color image and perform preprocessing including normalization; 2) Use the convolution module to reduce the channel dimension of the preprocessed infrared pseudo-color image to grayscale; 3) After flattening and batch normalizing the grayscale image, a preliminary temperature matrix is ​​calculated using a fully connected module; 4) The initial temperature matrix is ​​post-processed using the post-processing module, including inverse standard normalization, to obtain the final temperature matrix; In step 2), when the convolution module is used to reduce the channel dimension of the preprocessed infrared pseudo-color image to grayscale, the convolution module contains one or more convolutional layers. Each convolutional layer is a convolution with a kernel size of 1×1 followed by a ReLU activation function. The first convolutional layer of the convolution module has 3 input channels, and the last convolutional layer of the convolution module has 1 output channel. After the four-dimensional array M5 is processed by the convolution module, a four-dimensional array M6 with dimension [b, 1, a, a] is obtained as the grayscale image output after channel dimension reduction. In step 3), when the preliminary temperature matrix is ​​calculated using the fully connected module, the fully connected module contains one or more fully connected layers, and the input dimension of the first fully connected layer of the fully connected module is d, and the output dimension of the last fully connected layer of the fully connected module is d. After processing by the fully connected module, a two-dimensional array M9 with dimension [b, d] is obtained as the preliminary temperature matrix. Step 4) includes: 4.1) Perform a dimension transformation on the two-dimensional array M9, which represents the initial temperature matrix with dimensions [b, d], to obtain a three-dimensional array M with dimensions [b, a, a]. 10 ; 4.2) For the three-dimensional array M 10 Performing reverse concatenation yields a two-dimensional array M with dimensions [H, W]. 11 ; 4.3) Remove the two-dimensional array M 11 The surrounding padding results in a two-dimensional array M with dimensions [height, width]. 12 ; 4.4) For the two-dimensional array M 12 Performing inverse standard normalization yields the final temperature matrix M with dimensions [height, width]. 13 The relevant parameters for inverse normalization include the mean and variance.

2. The infrared image temperature matrix mapping method based on a nonlinear regression network according to claim 1, characterized in that, Step 1) includes: 1.1) Read the infrared pseudo-color image to obtain a three-dimensional array M1 with dimensions [3, height, width], where 3 is the color channel of the image, height is the height of the image, and width is the width of the image; 1.2) Standard normalization is performed on the three-dimensional array M1 to obtain a three-dimensional array M2 with dimensions [3, height, width]. The relevant parameters for standard normalization include mean and variance. 1.3) Calculate the remainder r of the image height with respect to the given parameter a. h The remainder r of the image width with respect to parameter a w If the remainder r h If not equal to 0, then fill ar symmetrically in the height direction. h With 1 pixel, fill values ​​M2[:, 0,:] and M2[:, -1, :] respectively, resulting in a 3D array M3 of dimension [3, H, width], where the height H satisfies H=height+ar h If the remainder r w If not equal to 0, then fill 'ar' symmetrically in the width direction. w With 1 pixel, fill values ​​M3[:, :, 0] and M3[:, 0, -1] respectively, resulting in a 3D array M4 with dimensions [3, H, W] after filling, where the width W satisfies W=width+ar w The three-dimensional array M4 is clipped to size a*a and then concatenated in the new dimensions to obtain a four-dimensional array M5 with dimensions [b, 3, a, a] as the output feature of the preprocessing, where the variable b = H / a * W / a.

3. The infrared image temperature matrix mapping method based on a nonlinear regression network according to claim 2, characterized in that, Step 3) involves flattening and batch normalizing the grayscale image, including: 3.1) The four-dimensional array M6 of the grayscale image output after channel dimensionality reduction with dimensions [b, 1, a, a] is flattened after retaining dimension 0, resulting in a two-dimensional array M7 with dimensions [b, d], where the variable d = 1*a*a; 3.2) Perform batch standard normalization on the two-dimensional array M7 to obtain a two-dimensional array M8 with dimensions [b, d].

4. The infrared image temperature matrix mapping method based on a nonlinear regression network according to claim 3, characterized in that, The function expression for batch standard normalization of the two-dimensional array M7 in step 3.2) is: In the above formula, For the second-dimensional array M8, the first... bs Batch No. l One element, For the second-dimensional array M7, the first... bs Batch No. l One element, As a scale factor, The mean of the two-dimensional array M7. Let M7 be the standard deviation of the two-dimensional array. For parameters close to 0, is the translation factor, and the mean, variance, scaling factor, and translation factor are the learning parameters of the nonlinear regression network.

5. The infrared image temperature matrix mapping method based on a nonlinear regression network according to claim 1, characterized in that, Step 1) also includes building and training a nonlinear regression network: S1. Prepare infrared pseudo-color image samples and their temperature matrices as labels to construct a training dataset; S2, construct a nonlinear regression network and initialize the network weight parameters of the nonlinear regression network; S3. Input the infrared pseudo-color images in the training dataset into the nonlinear regression network for forward computation and processing. Calculate the regression loss between the temperature matrix output by the nonlinear regression network and the temperature matrix used as the label. Based on the regression loss, calculate the gradient of the loss function with respect to the weights of neurons in each network layer and update the network weight parameters of the nonlinear regression network. Repeat the above steps until the specified number of training iterations are completed or the regression loss is less than the set value.

6. An infrared image temperature matrix mapping system based on a nonlinear regression network, comprising a microprocessor and a memory interconnected, characterized in that, The microprocessor is programmed or configured to perform the steps of the infrared image temperature matrix mapping method based on a nonlinear regression network as described in any one of claims 1 to 5.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that is programmed or configured to perform the infrared image temperature matrix mapping method based on a nonlinear regression network as described in any one of claims 1 to 5.

Images