Circuit infrared thermal image fault diagnosis method based on transmission transformation and convolutional neural network

Abstract

The present application belongs to the technical field of circuit fault detection, and discloses a circuit infrared thermal image fault diagnosis method based on transmission transformation and convolution neural network, which analyzes a target circuit, obtains a circuit thermal image, a visible light image under each fault condition of the circuit, and an output waveform image of the circuit in an oscilloscope; adopts an image transmission transformation method to perform heterogeneous fusion on the optical image, the infrared thermal image, and the oscilloscope image of the circuit under the same fault condition, to obtain a fused image of multiple images; after the fused image set is enhanced, the image set is divided into a training set and a verification set, and is sent into a convolution neural network model for training and classification, to verify the accuracy of the convolution neural network model. The accuracy of the model can reach 97%, which is improved by 3.4% compared with only using the infrared thermal image and the visible light image, is improved by 2.7% compared with a classic neural network LeNet5, has a smaller accuracy improvement compared with Vgg16, but the training time is shortened by 15 minutes, and the loss values are reduced by different amplitudes.

Description

Technical Field

[0001] The present invention belongs to the technical field of circuit fault detection, and in particular relates to a circuit infrared thermal image fault diagnosis method based on transmission transformation and convolutional neural network. Background Art

[0002] Currently, radio frequency circuits are widely used in electronics, computers, communications, aerospace, and other fields. Severe circuit failures can paralyze the entire system, leading to significant losses. However, with the increasing scale of circuits and the miniaturization of electronic components, traditional contact-based circuit fault detection methods are unable to meet the requirements of testing and maintaining increasingly complex electronic equipment. Thermal fault detection technology, however, has attracted widespread attention and application due to its non-contact, non-destructive, and rapid performance. Therefore, research on thermal fault detection technology holds significant economic significance.

[0003] The application of circuit infrared thermal image fault diagnosis technology based on transmission transform and convolutional neural networks is primarily reflected in fault feature extraction and fault type determination. Technological breakthroughs in artificial intelligence have brought new opportunities for fault diagnosis in the field of RF circuits. Its main advantage is the ability to effectively leverage existing data and deeply mine relevant hidden information. Convolutional neural networks, as one of the most representative neural networks in deep learning and widely used in image classification, therefore apply them to circuit infrared thermal image fault diagnosis based on transmission transform and convolutional neural networks to achieve significant practical applications.

[0004] Infrared thermal imaging fault diagnosis technology has been developed for many years. For example, in the United States and Canada, as early as the 1970s, Baker Hughes developed a system that used infrared equipment for isolation detection. However, in circuit board applications, the effect was not reflected due to the low performance of thermal imagers. With the rapid development of thermal detection technology, in 2021, scholars from the University of Agder, Norway, used drones equipped with thermal imagers and cameras to take composite images of solar panels in photovoltaic power plants under various fault conditions in the air, extracted fault features, and finally used convolutional neural networks to classify faults. Their main goal was to evaluate whether image processing of composite images can improve detection and classification performance compared to using thermal images alone.

[0005] China's infrared diagnostic technology started relatively late, with the earliest and most mature application being the "TIP (Thermal Image Processing) Fast Large-Area Non-Contact Circuit Fault Detector," developed by the University of Electronic Science and Technology of China in 2007. The TIP circuit fault detector is essentially an infrared detection device that collects infrared radiation images of a normal working circuit board and a circuit board under test under the same stimulus, performs correlation operations, and generates a circuit differential thermal map, or error image. This error image is then processed to locate the fault on the circuit board under test. Since then, Chinese scholars have rapidly expanded their research on thermal detection technology. In 2017, Wang Jiao of Xidian University, building on research on circuit board fault detection technologies using optical images and infrared thermal imaging, proposed a new non-contact electronic circuit thermal imaging anomaly detection method based on infrared thermal image sequences. The method's effectiveness has been verified in multiple experiments.

[0006] Radio frequency technology permeates every aspect of life. However, with the miniaturization of circuit components, traditional contact detection methods have become inadequate for troubleshooting electronic equipment. Infrared thermal imaging fault diagnosis, with its non-contact, non-destructive, and rapid nature, perfectly enables real-time monitoring of circuits. However, currently, relying solely on infrared thermal imaging can only locate the fault, not determine its cause.

[0007] Through the above analysis, the problems and defects of the existing technology are: the traditional contact circuit fault detection method can no longer meet the detection and maintenance problems of increasingly complex electronic equipment, and currently relying solely on infrared thermal diagnosis technology can only locate the fault but cannot determine the cause of the fault, that is, only surface information of the fault can be obtained but not necessarily the true cause of the fault. Summary of the Invention

[0008] In response to the problems existing in the prior art, the present invention provides a circuit infrared thermal image fault diagnosis method based on transmission transformation and convolutional neural network, and in particular relates to a radio frequency circuit heterogeneous fusion image fault diagnosis method, system, medium, equipment and terminal based on transmission transformation and convolutional neural network.

[0009] The present invention is implemented as follows: a circuit infrared thermal image fault diagnosis method based on transmission transformation and convolutional neural network includes: analyzing the target circuit to obtain thermal images, visible light images and output waveform images of the circuit in an oscilloscope under various fault conditions of the circuit; using the image transmission transformation method to heterogeneously fuse the optical image, infrared thermal image and oscilloscope image of the circuit under the same fault condition to obtain a fused image of multiple images; enhancing the fused image set and dividing it into a training set and a verification set, and then sending them to a convolutional neural network model for training and classification, and finally verifying the accuracy of the convolutional neural network model in fault recognition.

[0010] Furthermore, the circuit infrared thermal image fault diagnosis method based on transmission transformation and convolutional neural network includes the following steps:

[0011] Step 1: Obtain a visible optical image, an infrared thermal image, and an oscilloscope image of the circuit;

[0012] Step 2: Use the transmission transformation registration model to perform registration and fusion processing on the acquired images;

[0013] Step three: After image enhancement, the image set is divided into training set and validation set, the model parameters and evaluation indicators are determined, and the convolutional neural network model is trained and improved and optimized respectively.

[0014] In step 1, to validate this method, we selected multiple target circuit boards and assigned a variety of faults, then used the corresponding instruments to obtain visible light images, thermal images, and oscilloscope images for each fault condition. This experiment involved three circuit boards, each with similar performance parameters. We assigned 16 faults to each board and acquired three images for each condition.

[0015] In step 2, the image obtained in step 1 is registered and fused using an algorithm for converting any quadrilateral into a rectangle based on the transmission transformation registration model. Specifically, the algorithm includes:

[0016] (1) Read the image to be processed, select four points in a clockwise direction, in the order of upper left, upper right, lower right, and lower left, respectively, record the horizontal and vertical coordinates of the read points and use them as the vertices of the new rectangle; use the information of the selected points to calculate the optimal single mapping transformation matrix and perform a transmission transformation, so that only the pixels within the range of the four points remain in the new image through the transmission transformation;

[0017] (2) The transformed thermal map and visible light map are sized uniformly and fused at a ratio of 0.5:0.5 to obtain the circuit infrared thermal map and visible light fusion map under normal working conditions; the oscilloscope map is fused at a pixel value ratio of 0.35:0.35:0.3 to obtain the oscilloscope map, circuit infrared thermal map and visible light fusion map.

[0018] The image enhancement methods in step 3 include color to grayscale, channel separation, histogram equalization and image rotation. The image set includes three types of blue, green and red images after the original image channels are separated. Figure 1 The six images are divided into three groups: one group with histogram equalization and the original image. Taking into account the angle error during shooting, the six images are rotated within 5° left and right, with an interval of 0.5°.

[0019] The convolutional neural network model in step 3 is a model that is continuously improved and optimized through transfer learning and reference to the accuracy and loss rate of the training set and validation set during actual training. The optimized parameters include the learning rate, number of iterations, number of convolutional layers and convolution kernels, number of pooling layers and pooling kernels.

[0020] The evaluation indicators of model training include accuracy and loss rate. The accuracy is the number of correctly classified samples divided by the total number of samples; the loss rate is defined as the difference between the predicted value calculated by the network forward propagation and the standard value. The loss rate is calculated using mean square error and cross entropy.

[0021] A confusion matrix evaluation metric containing information about the predicted and actual types was constructed to verify the convolutional neural network model's ability to identify each type of fault. The deep learning networks LeNet5 and Vgg16 were used to compare the image training results.

[0022] Another object of the present invention is to provide a circuit infrared thermal image fault diagnosis system based on transmission transformation and convolutional neural network using the circuit infrared thermal image fault diagnosis method based on transmission transformation and convolutional neural network. The circuit infrared thermal image fault diagnosis system based on transmission transformation and convolutional neural network includes:

[0023] An image acquisition module, used to acquire optical images, infrared thermal images, and oscilloscope images of the circuit;

[0024] Image preprocessing module, used to perform image registration and fusion processing using a transmission transformation registration model;

[0025] The model training module is used to divide the image into training set and validation set after image enhancement, determine the model parameters and evaluation indicators, and train and improve the convolutional neural network model respectively.

[0026] In combination with the above technical solutions and the technical problems solved, the advantages and positive effects of the technical solutions to be protected by the present invention are as follows:

[0027] First, in view of the technical problems existing in the above-mentioned prior art and the difficulty of solving these problems, we closely combine the technical solutions to be protected by the present invention and the results and data during the research and development process, and conduct a detailed and in-depth analysis of how the technical solutions of the present invention solve the technical problems and some creative technical effects brought about by solving the problems. The specific description is as follows:

[0028] The present invention proposes a radio frequency circuit infrared thermal image fault diagnosis method based on image perspective transformation (IPT) and convolutional neural network (CNN). The present invention first performs heterogeneous fusion of the optical image, infrared thermal image and oscilloscope image of the circuit under the same fault condition, and then enhances the fused image set to divide it into a training set and a validation set, and finally sends it to the convolutional neural network for training and classification. The accuracy of the final validation set of the present invention can reach 97.3%, which is 3.4% higher than the accuracy of using only the fusion of infrared thermal image and visible light image.

[0029] At present, the problem of infrared fault diagnosis can be roughly divided into two directions. One is to diagnose faults in electronic power circuits or equipment, the main purpose of which is to locate the faults; the other is to use open source infrared thermal image collections in related fields, mainly to improve or innovate fault diagnosis methods, and improve the fault recognition rate of the improved method model. The radio frequency circuit infrared thermal image fault diagnosis method based on IPT and CNN based on transmission transformation and convolutional neural network proposed in the present invention aims to complete fault location while being able to more accurately determine the cause of the fault. The present invention first analyzes the target circuit, designs multiple faults under experimental conditions to ensure circuit safety, and uses the IPT method to obtain a fusion image of multiple images, and finally trains the CNN model and verifies the accuracy of the model.

[0030] Second, considering the technical solution as a whole or from the perspective of the product, the technical effects and advantages of the technical solution to be protected by the present invention are described in detail as follows:

[0031] Currently, research on infrared fault diagnosis in the field of radio frequency circuits is still in the exploratory stage. This paper proposes a novel circuit infrared thermal image fault diagnosis method based on IPT and CNN, using transmission transform and convolutional neural networks. Using circuit boards used in actual scientific research as research objects, a series of artificial faults were designed and infrared thermal images, visible light images, and oscilloscope images that represent these faults were extracted. These images were then integrated into a convolutional neural network model for training and validation using a method based on image transmission transform. The final model achieved an accuracy of approximately 97% on the validation set, a 3.4% improvement over using only infrared thermal images and visible light images, a 2.7% improvement over the classic neural network LeNet5, and a smaller improvement over Vgg16. However, the training time was shortened by approximately 15 minutes, and the loss values ​​were reduced to varying degrees. Experimental results demonstrate that the proposed circuit infrared thermal image fault diagnosis method based on transmission transform and convolutional neural networks is superior.

[0032] Third, as auxiliary evidence for the inventiveness of the claims of the present invention, it is also reflected in the following important aspects:

[0033] (1) The expected benefits and commercial value of the technical solution of the present invention after transformation are:

[0034] It can improve the efficiency of faulty circuit board classification, so as to know which fault occurs most frequently, and then prescribe the right remedy, improve circuit board design and fault identification methods, and reduce losses caused by sudden failures.

[0035] (2) The technical solution of the present invention fills the technical gap in the industry at home and abroad:

[0036] Currently, most infrared thermal diagnostic technologies at home and abroad only focus on the surface information of the fault, without conducting in-depth research. This paper proposes an infrared thermal image fault diagnosis method for RF circuits based on transmission transform and convolutional neural networks based on IPT and CNN, aiming to locate the fault while accurately determining the cause of the fault. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments of the present invention. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0038] Figure 1 This is a flow chart of a circuit infrared thermal image fault diagnosis method based on transmission transformation and convolutional neural network provided by an embodiment of the present invention;

[0039] Figure 2 This is a flowchart of an experiment provided by an embodiment of the present invention;

[0040] Figure 3A This is an infrared thermal image of a circuit under normal working conditions provided by an embodiment of the present invention;

[0041] Figure 3B is a visible light image under normal working conditions provided by an embodiment of the present invention;

[0042] Figure 3C is an oscilloscope waveform diagram under normal working conditions provided by an embodiment of the present invention;

[0043] Figure 4A This is an infrared thermal image of a circuit under fault 13 provided by an embodiment of the present invention;

[0044] Figure 4B is a visible light image under fault 13 condition provided by an embodiment of the present invention;

[0045] Figure 4C is an oscilloscope waveform diagram under fault 13 provided in an embodiment of the present invention;

[0046] Figure 5A This is an infrared thermal image transmission transformation positioning diagram under normal working conditions provided by an embodiment of the present invention;

[0047] Figure 5B This is a visible light image transmission transformation positioning diagram under normal working conditions provided by an embodiment of the present invention;

[0048] Figure 5C This is an oscilloscope transmission transformation positioning diagram under normal working conditions provided by an embodiment of the present invention;

[0049] Figure 6A This is an infrared thermal image of a circuit after transmission transformation under normal operation provided by an embodiment of the present invention;

[0050] Figure 6B This is a visible light image of a circuit after transmission transformation under normal operation provided by an embodiment of the present invention;

[0051] Figure 6C This is an oscilloscope diagram after transmission transformation under normal operation provided by an embodiment of the present invention;

[0052] Figure 7A This is a fusion diagram of a circuit heat map and a visible light map under normal operation provided by an embodiment of the present invention;

[0053] Figure 7B This is a fusion diagram of three graphs under normal working conditions provided by an embodiment of the present invention;

[0054] Figure 8A The accuracy of the infrared heat map and visible light map model training provided by the embodiment of the present invention;

[0055] Figure 8B It is the training loss rate of the infrared heat map and visible light map model provided by the embodiment of the present invention;

[0056] Figure 9A It is the accuracy of model training based on the fusion of three graphs provided by the embodiment of the present invention;

[0057] Figure 9B is the model training loss rate based on the fusion of three graphs provided in an embodiment of the present invention;

[0058] Figure 10A is a confusion matrix of a validation set based on infrared heat maps and visible light images provided by an embodiment of the present invention;

[0059] Figure 10B This is a confusion matrix of a validation set based on the fusion of three graphs provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0060] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0061] In response to the problems existing in the prior art, the present invention provides a circuit infrared thermal image fault diagnosis method based on transmission transformation and convolutional neural network. The present invention is described in detail below with reference to the accompanying drawings.

[0062] 1. Explanatory Examples In order to enable those skilled in the art to fully understand how to implement the present invention, this section provides an illustrative example that expands upon the technical solutions of the claims.

[0063] like Figure 1 As shown, the circuit infrared thermal image fault diagnosis method based on transmission transformation and convolutional neural network provided by the embodiment of the present invention includes the following steps:

[0064] S101, analyzing the target circuit to obtain a circuit heat map, a visible light map under various fault conditions of the circuit, and an output waveform of the circuit on an oscilloscope;

[0065] S102, using an image transmission transformation method to perform heterogeneous fusion of the optical image, infrared thermal image, and oscilloscope image of the circuit under the same fault condition to obtain a fused image of the multiple images;

[0066] S103, the fused image set is divided into a training set and a validation set after enhancement, and is sent to the convolutional neural network model for training and classification, and the accuracy of the convolutional neural network model is verified.

[0067] As a preferred embodiment, the circuit infrared thermal image fault diagnosis method based on transmission transformation and convolutional neural network provided in the embodiment of the present invention specifically includes the following steps:

[0068] Step 1, obtaining a visible optical image, an infrared thermal image, and an oscilloscope image of the circuit;

[0069] Step 2: Use the transmission transformation registration model to perform registration and fusion processing on the acquired images;

[0070] Step 3: After image enhancement, the image set is divided into a training set and a validation set, the model parameters and evaluation indicators are determined, and the convolutional neural network model is trained and improved and optimized respectively.

[0071] In step 1 provided in the embodiment of the present invention, three circuit boards are used as experimental objects. The performance parameters of each circuit board are similar and the experimental fault conditions are ensured to be the same. Finally, three images under the corresponding fault conditions are obtained.

[0072] In step 2 provided by the embodiment of the present invention, an algorithm for converting any quadrilateral into a rectangle based on a transmission transformation registration model is used to register and fuse the images obtained in step 1, specifically including:

[0073] (1) Read the image to be processed, select four points in a clockwise direction, in the order of upper left, upper right, lower right, and lower left, respectively, record the horizontal and vertical coordinates of the read points and use them as the vertices of the new rectangle; use the information of the selected points to calculate the optimal homography transformation matrix and perform a transmission transformation, so that only the pixels within the range of the four points remain in the new image through the transmission transformation;

[0074] (2) The transformed thermal map and visible light map are sized uniformly and fused at a ratio of 0.5:0.5 to obtain the circuit infrared thermal map and visible light fusion map under normal working conditions; the oscilloscope map is fused at a pixel value ratio of 0.35:0.35:0.3 to obtain the oscilloscope map, circuit infrared thermal map and visible light fusion map.

[0075] The image enhancement method in step 3 provided by the embodiment of the present invention includes color to grayscale conversion, channel separation, histogram equalization and image rotation. The image set includes three types of blue, green and red images after the original image channels are separated. Figure 1 The image set was divided into training and validation sets at a ratio of 2:1. The accuracy of the network model on the validation set was used to determine the accuracy of the fault determination.

[0076] The convolutional neural network model in step 3 provided in the embodiment of the present invention is a model that is continuously improved and optimized through transfer learning and reference to the accuracy and loss rate of the training set and validation set during actual training. The optimized parameters include the learning rate, the number of iterations, the number of convolutional layers and convolution kernels, the number of pooling layers and pooling kernels. The structure of the optimized convolutional neural network model includes:

[0077] Convolutional layer 1, output shape is (62, 62, 8), the number of parameters is 224;

[0078] Batch normalization layer 1, output shape is (62, 62, 8), the number of parameters is 32;

[0079] Pooling layer 1, output shape is (31, 31, 8), the number of parameters is 0;

[0080] Convolutional layer 2, output shape is (29, 29, 16), the number of parameters is 1168;

[0081] Batch normalization layer 2, with an output shape of (29, 29, 16) and 64 parameters;

[0082] Pooling layer 2, output shape is (14, 14, 16), the number of parameters is 0;

[0083] Convolutional layer 3, output shape is (12, 12, 32), the number of parameters is 4640;

[0084] Batch normalization layer 2, with an output shape of (12, 12, 32) and 128 parameters;

[0085] Pooling layer 2, output shape is (6, 6, 32), the number of parameters is 0;

[0086] Flatten layer, output shape is 1152, number of parameters is 0;

[0087] Fully connected layer 1, output shape is 128, number of parameters is 147584;

[0088] Dropout layer 1, output shape is 128, number of parameters is 0;

[0089] Fully connected layer 2, output shape is 64, number of parameters is 8256;

[0090] Dropout layer 2, output shape is 64, number of parameters is 0;

[0091] Fully connected layer 3, output shape is 16, number of parameters is 1040.

[0092] In step 3 provided in an embodiment of the present invention, the evaluation indicators of model training include accuracy and loss rate. The accuracy is the number of correctly classified samples divided by the total number of samples; the loss rate is defined as the difference between the predicted value calculated by the network forward propagation and the standard value. The calculation method of the loss rate includes mean square error and cross entropy.

[0093] A confusion matrix evaluation metric containing information about the predicted and actual types was constructed to verify the convolutional neural network model's ability to identify each type of fault. The deep learning networks LeNet5 and Vgg16 were used to compare the image training results.

[0094] The circuit infrared thermal image fault diagnosis system based on transmission transformation and convolutional neural network provided by the embodiment of the present invention includes:

[0095] An image acquisition module, used to acquire optical images, infrared thermal images, and oscilloscope images of the circuit;

[0096] Image preprocessing module, used to perform image registration and fusion processing using a transmission transformation registration model;

[0097] The model training module is used to divide the image into training set and validation set after image enhancement, determine the model parameters and evaluation indicators, and train and improve the convolutional neural network model respectively.

[0098] 2. Application Examples: In order to demonstrate the creativity and technical value of the technical solution of the present invention, this section provides application examples of the claimed technical solution on specific products or related technologies.

[0099] This paper proposes a circuit infrared thermal image fault diagnosis method based on IPT and CNN, using transmission transform and convolutional neural networks. Using circuit boards used in actual scientific research as research objects, a series of artificial faults were designed and the infrared thermal images, visible light images, and oscilloscope images that characterize these faults were extracted. These images were then integrated into a convolutional neural network model for training and validation using an image transmission transform. The resulting model achieved approximately 97% accuracy on the validation set, a 3.4% improvement over using only infrared thermal images and visible light images, a 2.7% improvement over the classic LeNet5 neural network, and a smaller improvement over Vgg16, but the training time was shortened by approximately 15 minutes, and the loss values ​​were reduced to varying degrees. Experimental results demonstrate that the proposed diagnostic method is superior.

[0100] 3. Evidence of the effects of the embodiments: The embodiments of the present invention have achieved some positive effects during the development or use process, and indeed have great advantages over the existing technology. The following content describes them with reference to the data, charts, etc. of the experimental process.

[0101] 1. Introduction to relevant basic theories

[0102] In nature, any object with a temperature above absolute zero radiates infrared waves. This property allows infrared thermal imagers to operate even in the pitch-black night. However, due to the pixel limitations of thermal imagers, extracting only infrared thermal images is limited and relatively simple. Therefore, this paper first fuses these images with higher-resolution optical images to form the image set for CNN model training.

[0103] 1.1 Image Heterogeneous Registration

[0104] Image registration is a typical problem in digital image processing. Based on its definition, image registration between images captured by the same device is often referred to as homologous registration, while registration between images captured by different devices is referred to as heterologous registration. Optical images have higher resolution but are significantly affected by the external environment; infrared thermal images are less affected by external lighting but have a lower pixel count. Combining these two can better characterize the fault characteristics under certain fault conditions. The image registration process can be mathematically expressed as Equation 1:

[0105] I2(X,Y)=I1(h(x,y)) (Equation 1)

[0106] Where I2(X, Y) is the grayscale value of the standard image, I1(x, y) is the grayscale value of the image to be registered, and h represents the two-dimensional geometric transformation function. Usually, the h function can be represented by rigid transformation, affine transformation, and transmission transformation models.

[0107] 1.2 Image Registration Model

[0108] In the field of image transformation, the most basic geometric transformations include translation, rotation, and scaling, also known as zooming. In practical applications, these transformations are often combinations or variations of two or more of these. Based on the complexity of the mathematical model, image transformations can be categorized into three types: rigid body transformation, affine transformation, and transmission transformation. Transmission transformation is used in this paper.

[0109] Transmission transformation, also known as projection transformation, only ensures that a straight line in the first image remains a straight line in the second image. From a physical perspective, transmission transformation is the process of projecting the first image onto another plane through a point. The shadow left on this plane is the transformed image. As shown in Equation 2:

[0110]

[0111] Among them, A 2×2 represents the affine transformation parameters, T 2×1 Represents the translation transformation parameter, V 1×2 Represents the projection transformation parameters, and s is a scaling factor related to the transformation parameters. From a formulaic perspective, a transmission transformation is also called a homography. The process of completing a projection transformation often involves calculating its homography matrix. This process can be thought of as transforming a two-dimensional image into another two-dimensional image in three-dimensional space. The present invention uses a transmission transformation to transform any quadrilateral within two images, thereby achieving heterogeneous registration.

[0112] 1.3 Convolutional Neural Networks

[0113] As a relatively mature model for deep learning, convolutional neural networks offer inherent advantages in feature extraction and modeling, particularly in image processing. Their local connections, shared weights, and pooling operations have overcome some of the challenges previously encountered in deep learning. With the significant growth of training datasets and the continuous improvement of hardware computing power, convolutional neural networks have gradually gained widespread attention. Convolutional neural networks typically consist of multiple convolutional layers, pooling layers, and fully connected layers. In practical applications, they often also include normalization and activation layers.

[0114] (1) Convolutional layer

[0115] A convolutional layer is generally composed of multiple feature planes and the same number of convolution kernels. Each feature plane is composed of multiple neurons and is connected to the previous layer through the convolution kernel. The convolution calculation formula is Equation 3:

[0116]

[0117] Where i represents the i-th convolution kernel, g(i) represents the feature map obtained by convolution of the i-th convolution kernel, a represents the pixel value of the input image, w and b represent the weight and bias of the convolution kernel respectively.

[0118] After convolution, an activation layer is added to activate the model to achieve nonlinearity. The most commonly used activation function is the ReLU activation function, which can effectively overcome the vanishing gradient problem.

[0119] (2) Pooling layer

[0120] The pooling layer typically follows the convolutional layer and is also composed of multiple feature planes. The pooling kernel is typically a 2×2 structure, designed to reduce the resolution of the feature plane to obtain spatially invariant features. This reduces the size of the feature plane in the pooling layer to 1 / 4 the size of the convolutional layer, allowing for secondary feature extraction and reducing the parameters of the convolutional neural network model. The most commonly used pooling method is max pooling, calculated as Equation 4:

[0121]

[0122] Among them, a l(i,t ) represents the t-th neuron in the i-th feature map in the l-th layer, w is the width of the convolution kernel, and j is the j-th pooling kernel.

[0123] (3) Fully connected layer

[0124] After multiple layers of convolution and pooling, a fully connected approach is needed to classify the feature faces. The parameters of the fully connected layer can account for over 80% of the parameters of the entire convolutional neural network model. Therefore, regularization methods, such as dropout, are often used in actual network models to mitigate overfitting. This technique temporarily removes a certain percentage of neurons from the hidden layer, preventing them from participating in forward and backward propagation, making the model more robust.

[0125] For multi-classification problems, fully connected layers typically use the Softmax function, which converts the output value of a neuron into its proportion among all neurons. This proportion can be considered the neuron's output probability. Currently, fully connected layers generally use regularization combined with the Softmax function to classify the output, achieving excellent classification performance.

[0126] 2. Experiment

[0127] This experiment can be divided into two parts: before and after the experiment. Before the experiment, you need to select other tools such as thermal imagers and RF circuit boards, and improve the experiment based on the difficulty and safety of actual operation; after the experiment, you need to select image processing and classification methods, and improve the methods or parameters based on the effect of image fusion and the accuracy of the classification model. The specific experimental process is as follows: Figure 2 shown.

[0128] 2.1 Experimental hardware selection

[0129] (1) Selection of infrared thermal imager

[0130] Among the many parameters of an infrared thermal imager, its infrared resolution, or pixels, is the primary determinant of its performance. Considering the size and number of chips and components on the circuit board, this experiment used a Shanghai Juge Infrared thermal imager with a resolution of 384×288 pixels, whose basic parameters meet the circuit board's requirements.

[0131] (2) Selection of experimental circuit board

[0132] Because the size and number of chips and components on a circuit board directly impact the recognizability of heat maps, this experiment selected a phase-locked loop (PLL) circuit board based on the AD4351 chip. This circuit outputs a sine wave frequency range of 35Hz to 300MHz, with phase noise less than 80dBc. The circuit includes an AD4351 PLL chip, the core chip responsible for frequency locking; an MSP430F1611 chip, which primarily controls the AD4351 PLL chip; a PL-2303HX chip for external program programming; and two 1117-5-3.3 power supply chips, each providing power to the PLL and 430 chips. In addition to the chip itself, there is also external filtering circuitry, including capacitors, inductors, and resistors. The circuit board size, chip dimensions, and component count all met the experimental requirements.

[0133] 2.2 Circuit Fault Settings

[0134] (1) PLL chip peripheral circuit fault setting:

[0135] Since short-circuit faults often result in excessive current and burn the chip, leading to high costs, the setting principle is to mainly detect open-circuit faults of inductors, resistors, and capacitors that determine key functions in the circuit. Each fault corresponds to a certain function of the chip pin. For example, a short-circuit fault of L3 corresponds to a loop filter failure, and a short-circuit fault of L4 corresponds to a voltage-controlled oscillator failure.

[0136] (2) 430 chip peripheral circuit fault settings:

[0137] Since the 430 chip has fewer peripheral circuits and only has a few passive crystal oscillator filter capacitors, only the capacitors are disconnected separately.

[0138] (3) Power chip fault setting:

[0139] Both power supply chips convert 5V to 3.3V and then supply power to the PLL chip and 430 chip respectively, so the fault type is basically the same, that is, the 5V or 3.3V pin of the chip is set to be open circuit fault.

[0140] Finally, considering the safety considerations of the circuit board and the operational difficulty during the actual experiment, the faults that can be set include 9 types of PLL chip peripheral circuit faults, 2 types of 430 chip peripheral circuit faults, 4 types of power chip faults, and the situation when the circuit is normal, a total of 16 situations, as shown in Table 1.

[0141] Table 1 Circuit fault condition number and description

[0142]

[0143] 2.3 Image Preprocessing

[0144] In addition to the thermal image of the circuit, the experiment of the present invention also took visible light images of the circuit under various fault conditions and measured the output waveform of the circuit in the oscilloscope. As long as it can reflect the characteristics of a certain fault condition of the circuit, it can be used to describe the fault. Due to the large number of fault types and image types, only the three images of normal conditions and fault 13 are compared; among them, under normal working conditions, the circuit infrared thermal image is as follows Figure 3A As shown, the visible light image is as follows Figure 3B As shown in the figure, the oscilloscope waveform is as follows Figure 3C As shown; under fault 13 conditions, the circuit infrared thermal image is as follows Figure 4A As shown, the visible light image is as follows Figure 4B As shown in the figure, the oscilloscope waveform is as follows Figure 4C shown.

[0145] As can be seen in the figure, the infrared heat map, visible light map, and oscilloscope map of the circuit board under fault condition 13 all differ. Ideally, a single heat map would be sufficient to characterize each fault condition. However, this is not always the case. In many cases, due to the integrity of the circuit, the failure of some components or functions has minimal impact on the heat generated by the circuit chip or component. Therefore, it is necessary to combine the circuit fault conditions under different instruments and equipment to characterize the circuit fault. The heat map reflects the thermal radiation information of the circuit board during the fault, the visible light map reflects the background information of the circuit board, and the oscilloscope map reflects the actual output information of the circuit board. Combining these three maps can provide a better and more comprehensive description of the fault.

[0146] In this case, since the images come from three different instruments and devices, the heterogeneous registration method of the present invention differs from the traditional registration method. In combination with the characteristics of the circuit board image, the present invention uses an arbitrary quadrilateral to rectangle algorithm based on the transmission transformation registration model to register and fuse the above images. The specific process is as follows:

[0147] (1) First, read the image to be processed, then select four points in a clockwise direction, in the order of upper left, upper right, lower right, and lower left. Record the horizontal and vertical coordinates of the read points and use them as the vertices of the new rectangle. Calculate the optimal homography transformation matrix using the information of the selected points and perform a transmission transformation. Only pixels within the range of the four points will remain in the new image through the transmission transformation.

[0148] Among them, under normal working conditions, the original infrared thermal image is as follows Figure 5A As shown, the original visible light image is as follows Figure 5B As shown, the original oscilloscope waveform is as follows Figure 5C As shown; Under normal working conditions, the infrared thermal image of the circuit after transmission transformation is as follows Figure 6A As shown, the circuit visible light diagram is as follows Figure 6B , the oscilloscope picture is as follows Figure 6C shown.

[0149] (2) Finally, the transformed thermal map and visible light map are unified in size and fused in a ratio of 0.5:0.5 to obtain Figure 7A Similarly, the pixel value ratio of 0.35:0.35:0.3 is also integrated with the oscilloscope graph to obtain Figure 7B .

[0150] 2.4 Convolutional Training Process

[0151] (1) Image enhancement

[0152] During the generation, transmission, or transformation of images, influenced by factors such as external light sources and noise, image quality may suffer from insufficient brightness and contrast. Therefore, image quality needs to be improved to better suit human vision or machine recognition. This experiment used three identical phase-locked loop circuit boards, two of which served as training objects and one as a validation object. Furthermore, neural network training requires a large number of images. To increase the number of images in the training set, image enhancement methods are required.

[0153] The enhancement methods used in the experiment of this invention are color to grayscale, channel separation, histogram equalization and image rotation. The image set includes three types of blue, green and red images after the original image channels are separated. Figure 1The network model then uses 1 type of histogram equalization, plus 1 type of original image, for a total of 6 types. Taking into account the angle deviation of the image capture, these 6 types of images are rotated left and right within 5°, with intervals of 0.5°. Ultimately, 360 images of each fault type are obtained, for a total of 16 types of faults. These images are divided into training and validation sets in a 2:1 ratio. The accuracy of the network model on the validation set is considered the accuracy of fault judgment.

[0154] (2) Model parameters and evaluation indicators

[0155] In convolutional neural network models, the parameters that primarily determine performance are model complexity and the learning rate. Model complexity calculations focus on hidden layers, which are layers between the input and output layers. Model complexity can generally be understood as a combination of temporal complexity and spatial complexity. The learning rate represents the learning capability of a convolutional neural network and, in the model, can be viewed as the magnitude of parameter updates during each iteration. In practical models, the settable learning rate is often an initial value; the actual learning rate adjusts based on the magnitude of parameter changes during iterations. The most commonly used learning rate is an exponentially decaying learning rate.

[0156] The neural network model used in this experiment was continuously improved and optimized through transfer learning and reference to the accuracy and loss rates of the training and validation sets during actual training. The main optimized parameters included the learning rate, number of iterations, number of convolutional layers and kernels, and number of pooling layers and kernels. The optimized convolutional neural network model structure is shown in Table 2.

[0157] Table 2 Optimized convolutional neural network model

[0158]

[0159] The quality of model training requires evaluation metrics, commonly used being accuracy and loss. Accuracy is the most common metric, calculated by dividing the number of correctly classified samples by the total number of samples. Generally speaking, higher accuracy indicates a better model, but in reality, correctly classified samples may be classified by a narrow margin. In such cases, the loss can more accurately describe the model's classification performance. The loss is defined as the difference between the predicted value and the standard value calculated by the network forward propagation. Its calculation depends on the loss function used by the model. Mainstream methods include mean square error (MSE) and cross entropy. These two methods have no significant difference when the predicted value is close to the standard value, but when the predicted value deviates significantly from the standard value, the mean square error penalizes it more severely. Cross entropy is often more effective in classification problems.

[0160] (3) Training results and comparison

[0161] To solve the problem of insufficient number of images, we use image enhancement; to solve the problem of low identification of certain faults, we use heat map combined with visible light map and oscilloscope map to improve it; to solve the problem of overfitting of convolutional neural network, we increase the number of image pixels and reduce the complexity of network model to alleviate it. In the recognition results of the final model, the model training accuracy is as follows: Figure 8A As shown, the model training loss rate is Figure 8B As shown in the infrared heat map, visible light map and oscilloscope map, the model training accuracy is as follows: Figure 9A As shown, the model training loss rate is Figure 9B shown.

[0162] As can be seen from the two figures, when the number of training times reaches 100, the accuracy and loss rate of the validation set no longer change significantly, indicating that the CNN model is basically trained. Figures 8A and 8B As shown in Figure 2, the final accuracy of fusing infrared thermal images and visible light images is 93.9%, and the loss value is 0.20; Figures 9A and 9B After adding the oscilloscope image, the final accuracy of the model on the validation set can reach 97.3%, and the loss rate is 0.09, both of which are improved. This shows that when the image set consists of three fusion images, the diagnosis results for 16 types of faults are more accurate.

[0163] In order to verify the recognition of each fault by the model of the present invention, the present invention provides a confusion matrix evaluation index containing relevant prediction and actual type information, such as Figures 10A and 10B As shown; Among them, the confusion matrix of the validation set, the two graphs are fused as shown Figure 10A As shown, the three graphs are fused as Figure 10B shown.

[0164] Fault types 1 to 16 correspond to F1 to F16, respectively. The x-axis represents the number of predicted fault types, and the y-axis represents the number of actual fault types. A total of 120 faults were verified for each type. The figure shows that when the image set is fused using the three types of images, the recognition rate for each fault type improves. For example, the recognition rate for faults 8 and 14 reaches 100% after adding the oscilloscope image. However, the validation sets for other fault types exhibit some error.

[0165] Table 3 compares the results of image training using two classic deep learning networks, LeNet5 and Vgg16. LeNet5 has a relatively low model complexity, while Vgg16 has a very high model complexity. The complexity of our model lies somewhere in between. As shown in Table 3, our model achieves the highest accuracy, reaching 97%, when using a three-graph fusion graph. LeNet5 has a lower model complexity, but higher accuracy and loss. Vgg16 has a higher accuracy, but takes a long time to iterate. In summary, our model performs better.

[0166] Table 3 Training results of different image sets

[0167]

[0168] It should be noted that the embodiments of the present invention can be implemented by hardware, software, or a combination of software and hardware. The hardware portion can be implemented using dedicated logic; the software portion can be stored in a memory and executed by an appropriate instruction execution system, such as a microprocessor or dedicated design hardware. Those skilled in the art will appreciate that the above-mentioned devices and methods can be implemented using computer-executable instructions and / or contained in processor control code, for example, such as a carrier medium such as a disk, CD or DVD-ROM, a programmable memory such as a read-only memory (firmware), or a data carrier such as an optical or electronic signal carrier. The devices and modules of the present invention can be implemented by hardware circuits such as very large-scale integrated circuits or gate arrays, semiconductors such as logic chips, transistors, or programmable hardware devices such as field programmable gate arrays, programmable logic devices, etc., can also be implemented by software executed by various types of processors, or can be implemented by a combination of the above-mentioned hardware circuits and software, such as firmware.

[0169] The above description is only a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications, equivalent substitutions and improvements made by any technician familiar with this technical field within the technical scope disclosed by the present invention and within the spirit and principles of the present invention should be covered by the scope of protection of the present invention.

Claims

1. A circuit infrared thermal image fault diagnosis method based on transmission transformation and convolutional neural network, characterized in that: The circuit infrared thermal image fault diagnosis method based on transmission transformation and convolutional neural network includes: analyzing the target circuit to obtain the circuit thermal image, visible light images under various fault conditions of the circuit, and the output waveform of the circuit in an oscilloscope; using the image transmission transformation method to heterogeneously fuse the optical image, infrared thermal image, and oscilloscope image of the circuit under the same fault condition to obtain a fused image of multiple images; after enhancing the fused image set, dividing it into a training set and a validation set, and then sending it to the convolutional neural network model for training and classification, and finally verifying the accuracy of the convolutional neural network model; The circuit infrared thermal image fault diagnosis method based on transmission transformation and convolutional neural network includes the following steps: Step 1: Obtain a visible optical image, an infrared thermal image, and an oscilloscope image of the circuit; Step 2: Use the transmission transformation registration model to perform registration and fusion processing on the acquired images; Step 3: After image enhancement, the image set is divided into a training set and a validation set, the model parameters and evaluation indicators are determined, and the convolutional neural network model is trained and improved and optimized respectively; In step 2, the image obtained in step 1 is registered and fused using an algorithm for converting any quadrilateral into a rectangle based on the transmission transformation registration model. Specifically, the algorithm includes: (1) Read the image to be processed, select four points in a clockwise direction, in the order of upper left, upper right, lower right and lower left, respectively, record the horizontal and vertical coordinates of the read points and use them as the vertices of the new rectangle; use the information of the selected points to calculate the optimal single mapping transformation matrix and perform a transmission transformation, so that only the pixels within the range of the four points remain in the new image through the transmission transformation; (2) The transformed thermal map and visible light map are sized uniformly and fused at a ratio of 0.5:0.5 to obtain the circuit infrared thermal map and visible light fusion map under normal working conditions; the oscilloscope map is fused at a pixel value ratio of 0.35:0.35:0.3 to obtain the oscilloscope map, circuit infrared thermal map and visible light fusion map.

2. The circuit infrared thermal image fault diagnosis method based on transmission transformation and convolutional neural network according to claim 1, characterized in that: The image enhancement methods in step 3 include color-to-grayscale conversion, channel separation, histogram equalization, and image rotation; the image set includes three images (blue, green, and red) after channel separation of the original image, one grayscale image, one histogram equalization image, and the original image, a total of six images; The six images were rotated left and right within 5°, with an interval of 0.5°. Finally, 360 images were obtained for each fault, with a total of 16 faults. The image set was divided into a training set and a validation set in a ratio of 2:

1. The accuracy of the network model validation set was used to determine the accuracy of the fault judgment.

3. The circuit infrared thermal image fault diagnosis method based on transmission transformation and convolutional neural network according to claim 1, characterized in that: The convolutional neural network model in step 3 is a model that is continuously improved and optimized through transfer learning and reference to the accuracy and loss rate of the training set and validation set during actual training. The optimized parameters include the learning rate, number of iterations, number of convolutional layers and convolution kernels, number of pooling layers and pooling kernels; Among them, the structure of the optimized convolutional neural network model includes: Convolutional layer 1, output shape is (62, 62, 8), the number of parameters is 224; Batch normalization layer 1, output shape is (62, 62, 8), the number of parameters is 32; Pooling layer 1, output shape is (31, 31, 8), the number of parameters is 0; Convolutional layer 2, output shape is (29, 29, 16), the number of parameters is 1168; Batch normalization layer 2, with an output shape of (29, 29, 16) and 64 parameters; Pooling layer 2, output shape is (14, 14, 16), the number of parameters is 0; Convolutional layer 3, output shape is (12, 12, 32), the number of parameters is 4640; Batch normalization layer 2, with an output shape of (12, 12, 32) and 128 parameters; Pooling layer 2, output shape is (6, 6, 32), the number of parameters is 0; Flatten layer, output shape is 1152, number of parameters is 0; Fully connected layer 1, output shape is 128, number of parameters is 147584; Dropout layer 1, output shape is 128, number of parameters is 0; Fully connected layer 2, output shape is 64, number of parameters is 8256; Dropout layer 2, output shape is 64, number of parameters is 0; Fully connected layer 3, output shape is 16, number of parameters is 1040.

4. The circuit infrared thermal image fault diagnosis method based on transmission transformation and convolutional neural network according to claim 1, characterized in that: In step 3, the evaluation indicators of model training include accuracy and loss rate. Accuracy is the number of correctly classified samples divided by the total number of samples. Loss rate is defined as the difference between the predicted value calculated by the network forward propagation and the standard value. The loss rate is calculated using mean square error and cross entropy. A confusion matrix evaluation metric containing information about the predicted and actual types was constructed to verify the convolutional neural network model's ability to identify each type of fault. The deep learning networks LeNet5 and Vgg16 were used to compare the image training results.

5. A circuit infrared thermal image fault diagnosis system based on transmission transformation and convolutional neural network, which uses the circuit infrared thermal image fault diagnosis method based on transmission transformation and convolutional neural network as described in any one of claims 1 to 4, characterized in that: The circuit infrared thermal image fault diagnosis system based on transmission transform and convolutional neural network includes: An image acquisition module, used to acquire optical images, infrared thermal images, and oscilloscope images of the circuit; Image preprocessing module, used to perform image registration and fusion processing using a transmission transformation registration model; The model training module is used to divide the image into training set and validation set after image enhancement, determine the model parameters and evaluation indicators, and train and improve the convolutional neural network model respectively.

Images