Infrared defect detection method, system and terminal based on thermal-optical flow spatio-temporal correction learning

Abstract

The application discloses an infrared defect detection method and system based on thermal-optical flow space-time correction learning and a terminal, belongs to the technical field of nondestructive testing, and comprises the following steps: acquiring thermal diffusion characteristic information maps and position characteristic information maps of defect regions; multi-level deep fusion decoding processing: directly superimposing the thermal diffusion characteristic information maps and the position characteristic information maps according to pixel point positions to obtain a first fusion feature map; superimposing the first fusion feature map with the thermal diffusion characteristic information maps and the position characteristic information maps respectively; taking the feature maps obtained by the two times of superimposition as inputs of the next level decoding; repeating the decoding processing steps to obtain final defect fusion information. The fusion processing can guide and correct the thermal diffusion information and the position information of the defects, so as to ensure the defect detection detection rate. Meanwhile, the feature information related to the defects is reserved and highlighted through the feature information map superimposition operation, the contrast of the defect regions is improved, and the defect detection accuracy is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of nondestructive testing, and in particular to an infrared defect detection method, system and terminal based on thermal-optical flow spatiotemporal correction learning. BACKGROUND

[0002] High quality and high safety are one of the hotspots in the process of material production and use. The industry usually uses nondestructive testing technology to detect the quality of materials. As a detection technology that does not damage the measured object, nondestructive testing plays an important role in improving product quality and ensuring the safe service of equipment structure.

[0003] Infrared thermal imaging detection technology is a common nondestructive testing technology, which further judges the physical characteristic information by obtaining the temperature information of the object. As one of the infrared thermal imaging detection technologies, the heating mode of optical excitation infrared thermal imaging (OPT) is to heat the object by an active light source. It has three excitation modes: pulse thermography (PT), lock-in thermography and pulse compression thermography. In the detection process, the thermal imager records the temperature change of the material, and the temperature change of different regions of the material is analyzed, so as to realize the rapid detection of defects of different depths in a large range, and become an important basis for material structure integrity evaluation and defect detection.

[0004] Due to the interference of environmental noise and heat diffusion, the collected thermal images have the defects of uneven heating, blurred edges, large environmental noise and weak defect information. There are many defect extraction processing methods that can improve the defect detection accuracy, such as traditional thermal imaging nondestructive testing algorithms: pulse phase method (PPT) and thermal signal reconstruction (TSR), and matrix tensor decomposition related algorithms: principal component analysis (PCA), independent component analysis (ICA), sparse integrated matrix decomposition method and low-rank tensor decomposition algorithm. The existing methods are to separate the components of the infrared data when detecting defects, and decompose the infrared defect thermal image sequence into background information, defect position information and noise information. According to this principle, the obvious defect area will be further highlighted, but for the not obvious defect area, it is difficult to improve the contrast of this area through component separation means, and even it may be classified as noise information and removed, so the existing methods have the problems of low defect detection rate and low defect contrast. SUMMARY

[0005] The purpose of the present application is to overcome the problems of the prior art and provide an infrared defect detection method, system and terminal based on thermal-optical flow spatiotemporal correction learning.

[0006] The purpose of the present application is achieved by the following technical solution: an infrared defect detection method based on thermal-optical flow spatiotemporal correction learning, which comprises the following steps:

[0007] Extract thermal diffusion characteristics of defect areas from infrared thermal images;

[0008] Calculate the location information of the defect area in the infrared thermal image;

[0009] Thermal diffusion feature information and location information are extracted to obtain thermal diffusion feature information map and location feature information map;

[0010] Multi-level deep fusion decoding process: The heat diffusion feature map and the position feature map are directly superimposed according to the pixel position to obtain the first fused feature map; the first fused feature map is superimposed with the heat diffusion feature map and the position feature map respectively to obtain the second fused feature map; the first fused feature map and the second fused feature map are used together as the input for the next level of decoding;

[0011] Repeat the above multi-level deep fusion decoding process to obtain the final defect fusion information.

[0012] In one example, the thermal diffusion characteristics of the defect region in the infrared thermogram are extracted using a thermo-optical flow algorithm, Fourier transform method, or frame interval difference method.

[0013] In one example, when using the thermo-optical flow algorithm to extract thermal diffusion feature information of the defect region in an infrared thermal image, the following are included:

[0014] Calculate the three-dimensional grayscale gradient values ​​of two infrared thermal images;

[0015] Calculate the lateral and longitudinal velocities of heat flow in two frames of infrared thermal images;

[0016] The thermal diffusion characteristics of the defect region are obtained by iteratively calculating the thermal optical flow based on the three-dimensional gray-scale gradient value, the transverse velocity of the thermal flow, and the longitudinal velocity of the thermal flow.

[0017] In one example, principal component analysis, independent component analysis, or L4 algorithm is used to calculate the location information of the defect region in the infrared thermogram.

[0018] In one example, when using the L4 algorithm to calculate the location information of the defect region in the infrared thermal image, the following steps are included:

[0019] The infrared thermal image sequence is processed by column vectorization frame by frame to construct the matrix corresponding to the infrared thermal image sequence;

[0020] Sparse representation of the matrix is ​​performed, and an optimized model of the sparse dictionary learning algorithm is established.

[0021] Set the number of iterations, solve the optimization model, and obtain the location information of the defect area.

[0022] In one example, the method further includes a fusion network setup step:

[0023] An encoder including at least one feature extraction layer is constructed to extract heat diffusion feature information and location information to obtain heat diffusion feature information map and location feature information map;

[0024] A decoder connected to the encoder is constructed. The decoder includes at least two sequentially connected decoding modules. Each decoding module includes parallel convolutional layers. The outputs of the convolutional layers are all connected to the fusion layer, and the outputs of the convolutional layers and the fusion layer are used as the inputs to the next level decoding module. The decoder is used to perform multi-level deep fusion decoding processing steps to obtain the final fused information.

[0025] In one example, the loss function L used during the training of the fusion network is:

[0026] L = L + L

[0027] sp

[0028] Among them, L s L represents the structural similarity loss. p This indicates the loss of pixel grayscale values.

[0029] It should be further noted that the technical features corresponding to the above examples can be combined or replaced to form new technical solutions.

[0030] This invention also includes an infrared defect detection system based on spatiotemporal correction learning of thermo-optical flow, which has the same inventive concept as the defect detection method formed by any or a combination of the above examples. The system includes:

[0031] The thermal diffusion feature information extraction unit is used to extract the thermal diffusion feature information of the defect area in the infrared thermal image;

[0032] The location information extraction unit is used to calculate the location information of the defect area in the infrared thermal image;

[0033] A converged network, comprising encoders and decoders connected in sequence;

[0034] The encoder includes at least one feature extraction layer for extracting heat diffusion feature information and location information to obtain heat diffusion feature information map and location feature information map;

[0035] The decoder includes at least two sequentially connected decoding modules. Each decoding module includes parallel convolutional layers, the outputs of which are all connected to a fusion layer. The outputs of the convolutional layers and the fusion layer are used as inputs to the next level decoding module. The decoder is used to perform multi-level deep fusion decoding processing steps to obtain the final fused information.

[0036] In one example, during the training of the fusion network, the decoder structure is replaced with:

[0037] The decoder includes at least one decoding layer, which is a convolutional layer.

[0038] It should be further noted that the technical features corresponding to the above system examples can be combined or replaced to form new technical solutions.

[0039] The present invention also includes a storage medium storing computer instructions that, when executed, perform the steps of the infrared defect detection method based on thermo-optical flow spatiotemporal correction learning formed by any or more of the above examples.

[0040] The present invention also includes a terminal comprising a memory and a processor, wherein the memory stores computer instructions executable on the processor, and the processor executes the steps of the infrared defect detection method based on thermo-optical flow spatiotemporal correction learning formed by any or more of the above examples when executing the computer instructions.

[0041] Compared with the prior art, the beneficial effects of the present invention are:

[0042] This invention first obtains a feature information map through feature extraction, and then fuses the thermal diffusion feature information map and the location feature information map. The fusion process enables the thermal diffusion information and location information of the defect to guide each other for correction, thereby ensuring the defect detection rate. At the same time, compared with direct fusion processing, it can effectively reduce the amount of data computation and significantly improve the data processing speed of the model. Furthermore, by superimposing the feature information map, the feature information related to the defect is retained and highlighted, which improves the contrast of the defect area and thus improves the accuracy of defect detection. Attached Figure Description

[0043] The specific embodiments of the present invention will be further described in detail below with reference to the accompanying drawings, which are used to provide a further understanding of the present application and constitute a part of the present application. The same reference numerals are used in these drawings to denote the same or similar parts. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation of the present application.

[0044] Figure 1 This is a flowchart of a method in an example of the present invention;

[0045] Figure 2 This is a schematic diagram of the defect thermal diffusion information extracted in an example of the present invention;

[0046] Figure 3 This is a schematic diagram of the defect location information extracted in an example of the present invention;

[0047] Figure 4 This is a schematic diagram of the final defect fusion information extracted in an example of the present invention;

[0048] Figure 5 This is the method framework corresponding to the preferred example of the present invention;

[0049] Figure 6 This is a schematic diagram of the fusion network structure during the training phase of this invention;

[0050] Figure 7 This is a schematic diagram of the fusion network structure used in the fusion processing of this invention. Detailed Implementation

[0051] The technical solution of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0052] In the description of this invention, it should be noted that the directions or positional relationships indicated by terms such as "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer" are based on the directions or positional relationships shown in the accompanying drawings. They are used only for the convenience of describing this invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention. Furthermore, the use of ordinal numbers (e.g., "first and second," "first to fourth," etc.) is for distinguishing objects and is not limited to this order, and should not be construed as indicating or implying relative importance.

[0053] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "joining" should be interpreted broadly. For example, they can refer to fixed connections, detachable connections, or integral connections; they can refer to mechanical connections or electrical connections; they can refer to direct connections or indirect connections through an intermediate medium; and they can refer to the internal communication between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0054] Furthermore, the technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0055] In one example, such as Figure 1 As shown, the infrared defect detection method based on thermal-optical flow spatiotemporal correction learning specifically includes the following steps:

[0056] S1: Extract thermal diffusion characteristics of defect areas from infrared thermal images;

[0057] S2: Calculate the location information of the defect area in the infrared thermal image;

[0058] S3: Perform fusion processing on defect thermal diffusion information and defect location information.

[0059] Optionally, thermal diffusion feature information of the defect area in the infrared thermogram can be extracted using a thermal optical flow algorithm, Fourier transform method, or frame interval difference method. Alternatively, thermal diffusion feature information of the infrared thermogram can be directly extracted based on a trained neural network model.

[0060] Optionally, the location information of the defect area in the infrared thermogram can be calculated using any matrix tensor decomposition algorithm such as PCA, independent component analysis, or L4.

[0061] Optionally, steps S1 and S2 can be executed simultaneously or in any order (execute step S1 first, then step S2; or execute step S2 first, then step S1).

[0062] Specifically, step S3, the fusion process, includes:

[0063] S31: Extract thermal diffusion feature information and location information to obtain thermal diffusion feature information map and location feature information map;

[0064] S32: The thermal diffusion feature map and the position feature map of the two channels are directly superimposed according to the pixel position to obtain the first fused feature map;

[0065] S33: Overlay the first fused feature map with the heat diffusion feature map and the location feature map respectively to obtain the second fused feature map; use the first fused feature map and the second fused feature map together as the next level decoding input φ. out ;

[0066] S34: Repeat steps S32-S33 above until the final defect fusion information is obtained. This invention, through multi-level fusion processing, achieves deeper fusion of feature information from various dimensions of the defect region, efficiently utilizing the features contained in different types of defect information, thereby improving subsequent defect detection performance.

[0067] Specifically, the thermal diffusion characteristic information and location information are input into the encoder for feature extraction. This information can be input into the encoder of the DenseFuse model for feature extraction, thereby obtaining the thermal diffusion characteristic information map φ of the defect. loc Location feature information map φ heat The feature map obtained after the thermal diffusion characteristic information and location information are compressed by the encoder contains high-dimensional features of the data. At the same time, the encoding and compression process solves the problem of redundant noise in infrared data.

[0068] Specifically, each level of decoding outputs a fused feature map φ out The formula for calculation is:

[0069] φ out (x,y)=φ loc (x,y)+φ heat (x,y)

[0070] In the formula, x and y represent the horizontal and vertical coordinate values ​​in the feature map. This invention first obtains a feature information map through feature extraction, and then fuses the thermal diffusion feature information map and the location feature information map. The fusion process enables the thermal diffusion information and location information of the defect to guide each other for correction, thereby ensuring the defect detection rate. At the same time, compared with direct fusion processing, it can effectively reduce the amount of data computation and significantly improve the data processing speed of the model. Furthermore, by superimposing the feature information map, the feature information related to the defect is retained and highlighted, which improves the contrast of the defect area and thus improves the defect detection accuracy.

[0071] In one example, a thermo-optical flow algorithm is used to extract thermal diffusion feature information of the defect region in the infrared thermal image, including:

[0072] S11: Calculate the three-dimensional grayscale gradient values ​​of the two infrared thermal images; specifically, read two frames of infrared defect image data and calculate the three-dimensional grayscale gradient I according to the following formula. x I y and I t :

[0073]

[0074]

[0075]

[0076] Where i, j, and k represent the pixel positions in the i-th row and j-th column of the k-th frame infrared thermal image, respectively.

[0077] S12: Calculate the lateral and longitudinal velocities of heat flow in the two infrared thermal images; specifically, calculate the mean values ​​of the lateral velocity u and the longitudinal velocity v using the nine-point difference method. and

[0078]

[0079]

[0080] S13: The thermo-optical flow is iteratively calculated based on the three-dimensional grayscale gradient value, the transverse velocity of the heat flow, and the longitudinal velocity of the heat flow to obtain the thermal diffusion characteristic information of the defect region. Specifically, an upper limit N for the number of iterations is set, and the thermo-optical flow is calculated iteratively; in the nth iteration:

[0081]

[0082] In the formula, I x I y I t The calculation method is shown in step S11. The calculation method is shown in step S12. λ is a small positive number used to prevent the denominator on the right side of the equation from being zero. After N iterations of the above formula, the final horizontal u can be obtained. N Longitudinal optical flow v N The arithmetic square root of these two This refers to information about the thermal diffusion of defects, such as... Figure 2 As shown.

[0083] In one example, the L4 algorithm is used to calculate the location information of the defect region in the infrared thermal image, including:

[0084] S21: Perform column vectorization on the infrared thermal image sequence frame by frame to construct the matrix corresponding to the infrared thermal image sequence; specifically, for each frame of data of size m×n×f, take the pixel value of each pixel in each row, then arrange them vertically in the order of value taking, and finally arrange the vertically arranged infrared thermal images of each frame in sequence to construct a new matrix and transpose it. The transposed matrix is ​​denoted as Y∈R. f×p :

[0085]

[0086]

[0087] Y = DX, D T D = I, X ~ iid BG(θ)

[0088] Where p has a size of m×n.

[0089] S22: The matrix is ​​sparsely represented as Y = DX, and an optimized model for the sparse dictionary learning algorithm is established as follows:

[0090]

[0091] Y = DX, D T D = I, X ~ iid BG(θ)

[0092] Where, D∈R f×kIt is a dictionary matrix; X∈R k×p It is a sparse coefficient matrix; k represents the eigenvalue; ||·||4 denotes the L4 norm; ||·|| F Denotes the Frobenius norm; X ~ iid BG(θ) indicates that the sparse coefficient matrix X follows a Bernoulli-Gaussian distribution; θ is a coefficient used to control the sparsity of the X matrix; P∈SP(k) represents the sign permutation matrix; matrix A∈R k×f It is a global maximizer, I∈R k×k It is the identity matrix; C and ε are threshold parameters.

[0093] S23: Set the number of iterations and solve the optimization model, which includes the following steps:

[0094] S231: Set the number of iterations q, q = 1, 2, ..., Q; Q represents the maximum number of iterations; initialize A0 = [eye(k,k); zeros(fk,k).

[0095] S232: For the formula Find A q The derivative is obtained As shown in the following formula:

[0096]

[0097] Here, "*" represents the transpose of the matrix.

[0098] S233: Yes The SVD decomposition is performed as follows:

[0099]

[0100] Where SVD(·) represents singular value matrix decomposition; U∈R k×k It is a left singular matrix; V∈R f×f It is a right singular matrix; Σ∈R k×f It is a singular value matrix.

[0101] S234: Let A q+1 The value is updated to:

[0102] A q+1 =UV *

[0103] S235: Number of update iterations q←q+1;

[0104] S236: If the number of iterations q = Q, then the iteration stops, A Q Y represents the algorithm's result; A Q Y is inversely matrixed (the inverse process of matrix transformation), resulting in the image sequence E∈R. m×n×kThis refers to the defect location information, such as... Figure 3 As shown in the figure. The defect location information obtained in this example and the thermal diffusion feature information obtained in the previous example are fused together, and multi-level deep fusion decoding is performed to obtain the final defect fusion information output as shown in the figure. Figure 4 As shown in the image, this example uses the L4 algorithm to calculate the location information of the defect area in the infrared thermal image. Compared with the PCA algorithm, it can reduce computational complexity and can be implemented on lower-configuration devices to achieve rapid defect detection.

[0105] In one example, the method also includes a fusion network setup step:

[0106] An encoder is constructed that includes at least one feature extraction layer to extract thermal diffusion feature information and location information to obtain thermal diffusion feature information map and location feature information map; wherein the feature extraction layer is a convolutional layer or a deep convolutional layer.

[0107] A decoder connected to the encoder is constructed. The decoder includes at least two sequentially connected decoding modules. Each decoding module includes parallel convolutional layers. The outputs of the convolutional layers are all connected to the fusion layer, and the outputs of the convolutional layers and the fusion layer are used as the inputs to the next level decoding module. The decoder is used to perform multi-level deep fusion decoding processing steps to obtain the final fused information.

[0108] In one example, the loss function L used during the training of the fusion network is:

[0109] L = L s +L p

[0110] Among them, L s L represents the structural similarity loss. p This represents the pixel grayscale value loss. Specifically, the formulas for calculating structural similarity loss and pixel grayscale value loss are as follows:

[0111]

[0112] L p =||OI||2

[0113] Where O and I represent the output and input images, respectively; SSIM(·) represents structural similarity calculation, and the closer the calculation result is to 1, the more similar the output and input images are in terms of global and local structure. Therefore, in the neural network calculation process, 1 minus SSIM(O, I) is used to represent the degree of dissimilarity between two images; μ I It is the average value of I; μ O It is the average value of O; It is the variance of I; σ is the variance of O; IOIt is the covariance of I and O; c1 = (k1L) 2 c1 = (k2L) 2 This is a constant used to balance the numerator and denominator; L is the dynamic range of I and O pixel values, and in this paper, k1 = 0.01 and k2 = 0.03 are taken; ||.||2 represents the L2 norm of the image pixel grayscale values. The smaller the value, the smaller the difference between the grayscale values ​​of the input and output images, that is, the closer the input and output images are. The fusion network requires a large amount of data to support it, in order to avoid overfitting and underfitting. This method uses the large-scale image dataset MS-COCO as the training dataset, with a total of 80,000 images.

[0114] Combining the above methods yields a preferred example of the present invention, such as... Figure 5 As shown, the defect detection method at this time includes the following steps:

[0115] S1': Build a fusion network and pre-train the network model;

[0116] S2': The thermal diffusion feature information of the defect region in the infrared thermal image is extracted using the thermal optical flow algorithm;

[0117] S3': The L4 algorithm is used to calculate the location information of the defect area in the infrared thermal image;

[0118] S4': The encoder is used to fuse the defect thermal diffusion information and defect location information;

[0119] S5': Use the decoder to perform multi-level deep fusion decoding processing to obtain the final defect fusion information.

[0120] This invention first uses a thermo-optical flow algorithm to fit and calculate the thermal diffusion information of the defect region, then uses the L4 matrix factorization algorithm to process the infrared data to obtain the location information of the defect region, and finally uses a spatiotemporal correction fusion network model to fuse the two different types of defect information at the feature level to solve the final comprehensive feature information map of the defect region, thereby realizing defect detection by infrared thermal imaging and improving the defect detection rate.

[0121] To verify the feasibility of the present invention, defect detection experiments were conducted using infrared defect data of composite materials. The F-score and SNR evaluation metrics were used to evaluate the detection performance of this method, where the F-score is defined as follows:

[0122]

[0123] Where β is the weighting of recall and precision, and its value is 2; Precision represents precision; Recall represents recall, defined by the following formulas:

[0124]

[0125]

[0126] In the formula, TP, FP, and FN represent true positives, false positives, and false negatives, respectively; true positives indicate that the defects detected in the processing results are real defect locations; false positives indicate that the defects detected in the processing results are actually non-defect locations; and false negatives indicate the missed detection of defect areas.

[0127] The contrast between defective and non-defective areas can be evaluated using SNR, defined as follows:

[0128]

[0129] Where Td represents the sum of pixel values ​​within the defective region, and Tnon represents the sum of pixel values ​​within the non-defective region.

[0130] In this embodiment, two types of specimens were used for the experiment: specimen 1 was a carbon fiber material specimen, and specimen 2 was a coating material specimen. Figure 2 , Figure 3 The results of various algorithmic processing of infrared data corresponding to the two types of specimens are shown below. Figure 2 This is the result of the thermal optical flow algorithm. Figure 3 This is the result of processing by the L4 matrix factorization algorithm. Figure 4 These are the processing results of the algorithm described in this invention. As can be seen from the various result images, the algorithm of this invention has a very good enhancement effect on the display of defect information in thermal images. Compared with other algorithms, it detects more defects, has higher defect contrast, and less image noise.

[0131] Table 1 shows a comparison of the evaluation metrics results of the thermal-optical flow algorithm, the L4 matrix factorization algorithm, and the spatiotemporal correction network fusion results:

[0132] Table 1 Comparison of Evaluation Indicator Results

[0133]

[0134] As shown in Table 1, for the F-score evaluation metric, the thermo-optical flow algorithm and the algorithm proposed in this invention have the highest F-score, 1.00. The L4 algorithm has a false negative defect detection rate, with a score of only 0.79. For the SNR evaluation metric, the algorithm proposed in this invention has the highest SNR, 2.86 and 2.93 respectively. Considering both evaluation metrics, the algorithm proposed in this invention has the highest SNR between the defect and non-defect regions while ensuring no false negatives or missed detections.

[0135] This embodiment also provides a storage medium that has the same inventive concept as the infrared defect detection method based on thermo-optical flow spatiotemporal correction learning formed by any or more of the above examples, and stores computer instructions thereon. When the computer instructions are executed, they perform the steps of the infrared defect detection method based on thermo-optical flow spatiotemporal correction learning formed by any or more of the above examples.

[0136] Based on this understanding, the technical solution of this embodiment, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0137] This application also includes a terminal having the same inventive concept as any or a combination of examples corresponding to the above-described infrared defect detection method based on thermo-optical flow spatiotemporal correction learning, including a memory and a processor. The memory stores computer instructions executable on the processor, which, when executing the computer instructions, performs the steps of the above-described infrared defect detection method based on thermo-optical flow spatiotemporal correction learning. The processor may be a single-core or multi-core central processing unit or a specific integrated circuit, or one or more integrated circuits configured to implement the present invention.

[0138] In one example, the terminal, i.e., the electronic device, is manifested in the form of a general-purpose computing device. The components of the electronic device may include, but are not limited to: at least one processing unit (processor) mentioned above, at least one storage unit mentioned above, and a bus connecting different system components (including storage units and processing units).

[0139] The storage unit stores program code that can be executed by the processing unit, causing the processing unit to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of the present invention. For example, the processing unit can execute the infrared defect detection method based on thermo-optical flow spatiotemporal correction learning described above.

[0140] The storage unit may include a readable medium in the form of a volatile storage unit, such as a random access memory (RAM) 3201 and / or a cache storage unit, and may further include a read-only memory (ROM).

[0141] The storage unit may also include a program / utility having a set (at least one) of program modules, including but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0142] A bus can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus that uses any of the various bus structures.

[0143] The electronic device can also communicate with one or more external devices (e.g., keyboards, pointing devices, Bluetooth devices, etc.), one or more devices that enable a user to interact with the electronic device, and / or any device that enables the electronic device to communicate with one or more other computing devices (e.g., routers, modems, etc.). This communication can be performed via input / output (I / O) interfaces. Furthermore, the electronic device can communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter. The network adapter communicates with other modules of the electronic device via a bus. It should be understood that other hardware and / or software modules can be used in conjunction with the electronic device, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0144] Through the above description, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to this exemplary embodiment can be embodied in the form of a software product, which can be stored on a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, terminal device, or network device, etc.) to execute the method of the exemplary embodiment of this application.

[0145] The present invention also includes an infrared defect detection system based on thermo-optical flow spatiotemporal correction learning, which has the same inventive concept as the defect detection method formed by any or more of the above examples. The system includes a thermal diffusion feature information extraction unit, a location information extraction unit, and a fusion network (correction network).

[0146] Among them, the thermal diffusion feature information extraction unit is used to extract the thermal diffusion feature information of the defect area in the infrared thermal image; the location information extraction unit is used to calculate the location information of the defect area in the infrared thermal image.

[0147] The fusion network includes an encoder and a decoder connected in sequence; the encoder includes at least one feature extraction layer for extracting thermal diffusion feature information and location information to obtain thermal diffusion feature information map and location feature information map; the feature extraction layer is a convolutional layer or a deep convolutional layer.

[0148] The decoder includes at least two sequentially connected decoding modules. Each decoding module includes parallel convolutional layers. The outputs of the convolutional layers are all connected to a fusion layer. The outputs of the convolutional layers and the fusion layer are used as the inputs of the next-level decoding module. That is, the outputs of the convolutional layers and the fusion layer are connected to the inputs of the parallel convolutional layers of the next-level decoding module. The decoder is used to perform multi-level deep fusion decoding processing steps and finally outputs the final fused information through a convolutional layer.

[0149] Optionally, such as Figures 6-7 As shown, the encoder includes a first convolutional layer C1, a first depthwise convolutional layer DC1, a second depthwise convolutional layer DC2, and a third depthwise convolutional layer DC3 connected in sequence. The decoder includes a first decoding module DE1, a second decoding module DE2, a third decoding module DE3, and a fifth convolutional layer C5 connected in sequence. More specifically, the detailed network parameters of the fusion network are shown in Table 2:

[0150] Table 2. Parameters of the Fusion Network

[0151]

[0152] In one example, during the training of the fusion network, such as Figure 5 As shown, the decoder structure is replaced with:

[0153] The decoder includes at least one decoding layer, which is a convolutional layer. In this case, only the single-channel decoding module structure needs to be trained, and each decoding module is reused during the fusion process (the parameters will not change after training), which can reduce the training workload.

[0154] The above detailed embodiments are a description of the present invention. It should not be considered that the specific embodiments of the present invention are limited to these descriptions. For those skilled in the art, several simple deductions and substitutions can be made without departing from the concept of the present invention, and all of these should be considered to fall within the protection scope of the present invention.

Claims

1. An infrared defect detection method based on spatiotemporal correction learning of thermo-optical flow, characterized in that: Includes the following steps: Extract thermal diffusion characteristics of defect areas from infrared thermal images; Calculate the location information of the defect area in the infrared thermal image; Thermal diffusion feature information and location information are extracted to obtain thermal diffusion feature information map and location feature information map; The thermal diffusion feature map and the location feature map are directly superimposed according to the pixel position to obtain the first fused feature map; the first fused feature map is superimposed with the thermal diffusion feature map and the location feature map respectively to obtain the second fused feature map. The method also includes a fusion network construction step: An encoder including at least one feature extraction layer is constructed to extract heat diffusion feature information and location information to obtain heat diffusion feature information map and location feature information map; A decoder connected to the encoder is constructed. The decoder includes at least two sequentially connected decoding modules. The heat diffusion feature information map and the position feature information map are input to the first-level decoding module. The decoding module includes parallel convolutional layers. The outputs of the convolutional layers are all connected to the fusion layer. The outputs of the convolutional layers and the fusion layer are used as the inputs of the next-level decoding module. The decoder is used to perform multi-level deep fusion decoding processing steps to obtain the final fused information; When using the thermo-optical flow algorithm to extract thermal diffusion feature information of defect regions in infrared thermal images, the following are included: Calculate the three-dimensional grayscale gradient values ​​of two infrared thermal images; Calculate the lateral and longitudinal velocities of heat flow in two frames of infrared thermal images; The thermal diffusion characteristics of the defect region are obtained by iteratively calculating the thermal optical flow based on the three-dimensional gray-scale gradient value, the transverse velocity of the thermal flow, and the longitudinal velocity of the thermal flow.

2. The infrared defect detection method based on spatiotemporal correction learning of thermo-optical flow according to claim 1, characterized in that: The location information of the defect area in the infrared thermogram is calculated using the principal component analysis algorithm, the independent component analysis algorithm, or the L4 algorithm.

3. The infrared defect detection method based on spatiotemporal correction learning of thermo-optical flow according to claim 2, characterized in that: When using the L4 algorithm to calculate the location information of the defect region in the infrared thermal image, it includes: The infrared thermal image sequence is processed by column vectorization frame by frame to construct the matrix corresponding to the infrared thermal image sequence; Sparse representation of the matrix is ​​performed, and an optimized model of the sparse dictionary learning algorithm is established. Set the number of iterations, solve the optimization model, and obtain the location information of the defect area.

4. The infrared defect detection method based on spatiotemporal correction learning of thermo-optical flow according to claim 3, characterized in that: The loss function L used in the training process of the fusion network is: ； Among them, L s L represents the structural similarity loss. p This indicates the loss of pixel grayscale values.

5. An infrared defect detection system based on thermo-optical flow spatiotemporal correction learning that performs the method as described in claim 1, characterized in that: It includes: The thermal diffusion feature information extraction unit is used to extract the thermal diffusion feature information of the defect area in the infrared thermal image; The location information extraction unit is used to calculate the location information of the defect area in the infrared thermal image; A converged network, comprising encoders and decoders connected in sequence; The encoder includes at least one feature extraction layer for extracting heat diffusion feature information and location information to obtain heat diffusion feature information map and location feature information map; The decoder includes at least two sequentially connected decoding modules. Each decoding module includes parallel convolutional layers. The outputs of the convolutional layers are all connected to a fusion layer, and the outputs of the convolutional layers and the fusion layer are used as the inputs to the next level decoding module. The decoder is used to perform multi-level deep fusion decoding processing steps to obtain the final fused information.

6. A terminal comprising a memory and a processor, wherein the memory stores computer instructions executable on the processor, characterized in that: When the processor executes the computer instructions, it performs the steps of the infrared defect detection method based on thermo-optical flow spatiotemporal correction learning as described in any one of claims 1-4.

Images