Robust principal component analysis-based pulse thermal imaging sequence key frame extraction method

Abstract

The invention discloses a pulse thermal imaging sequence key frame extraction method based on robust principal component analysis, and the method comprises the steps: firstly calculating a difference image between adjacent images of an original thermal image sequence, solving the L1 norm of a difference result, and determining a starting frame of pulse excitation according to the maximum value of the L1 norm; decomposing the thermal image sequence by using robust principal component analysis, and extracting a key frame according to the sparsity of a sparse matrix; and finally, extracting a sample region in the thermal image by using a local maximum entropy threshold segmentation method to obtain a thermal image key frame sequence containing the sample region. According to the method, manual key frame screening and ROI clipping in the prior art can be perfectly replaced, the space-time distribution characteristics of the thermal imaging sequence are fully utilized, a complete and efficient pulse thermal imaging sequence key frame automatic extraction algorithm framework is developed, and the defect recognition efficiency and the automation level are improved in practical application.

Description

Technical Field

[0001] The invention relates to an automatic extraction method of pulse thermal imaging sequence key frames for internal defect detection of carbon fiber composite materials, belonging to the technical field of non-destructive detection. Background Art

[0002] Carbon fiber composite materials are widely used in various high-end manufacturing fields due to their light weight, high strength and corrosion resistance. Their application has not only promoted technological progress in key fields such as national defense, aerospace, energy and industry, but also improved product performance and user experience in civilian fields such as transportation, construction, sports and medical care. However, during the manufacturing process, carbon fiber composite materials may have problems such as improper curing temperature control, stress concentration of plies, impurity mixing, and uneven resin distribution. The material is prone to internal defects such as holes, fiber breakage, debonding, and delamination. These defects will not only affect its mechanical properties, but may also have a serious impact on the safety, reliability and economy of the material, limiting the application of carbon fiber composite materials.

[0003] At present, non-destructive testing technology is widely used in the detection of internal defects in carbon fiber composite materials, and plays an important role in material quality control and safety inspection. As an advanced non-destructive testing method, pulse thermal imaging technology has the advantages of low detection cost, fast detection speed, and large-area detection. This technology uses a high-energy heat source to instantly heat the surface of the object being tested, and then uses an infrared thermal imager to capture the temperature changes on the surface of the material during the cooling process. Since internal defects in the material will affect the rate of heat propagation, the surface temperature of the material corresponding to the defect will be abnormally distributed. Therefore, defects can be identified and located by analyzing the thermal image of the material surface.

[0004] Through pulse thermal imaging detection, a thermal image sequence that expresses the dynamic changes in the material surface temperature can be obtained, and defect detection can be performed on this basis. The sequence contains four stages of material surface temperature changes, namely the thermal equilibrium stage before excitation, the moment of excitation, the dynamic heat transfer stage, and the thermal equilibrium stage after excitation. Among them, only the thermal image of the dynamic heat transfer process can fully reflect the defect information. In addition, the original thermal image contains both the sample and the surrounding environment, and the sample information needs to be further extracted as the region of interest (ROI) for subsequent detection. In summary, the key frame extraction and ROI extraction of the heat transfer process in the thermal image sequence are necessary preprocessing steps to ensure the implementation of defect detection. However, the detection algorithms currently proposed all require manual key frame screening and ROI cropping, which has high labor costs and low accuracy ((1) Liu W, Zhao P, Zhao Y, et al. A reflectance-correction retinex framework for thermal image enhancement in nondestructive defect detection of CFRP[J]. Measurement, 2024: 115070. (2) Wu JY, Sfarra S, Yao Y. Sparse principal component thermography for subsurface defect detection in composite products[J]. IEEE transactions on industrial informatics, 2018, 14(12): 5594-5600. (3) Liu Y, Wu JY, Liu K, et al. Independent component thermography for non-destructive testing of defects in polymer composites[J]. Measurement Science and Technology, 2019, 30(4): 044006.). Therefore, automatic extraction of thermal image key frames and ROI is of great significance to improving the efficiency of defect detection and enhancing the automation level of pulse thermal imaging nondestructive testing.

[0005] Key frame extraction algorithms play an important role in image sequence analysis. The current mainstream algorithms are mainly based on clustering, motion analysis, and frame difference methods. By extracting features such as color, texture, gradient, and optical flow, frame classification and extraction can be achieved. On the other hand, Robust Principal Component Analysis (RPCA), as a data-driven algorithm, does not require manual feature definition and can combine the internal structure information of the data to achieve data compression and effective information extraction.

[0006] At the same time, segmentation algorithms based on threshold segmentation, edge information method and deep learning can effectively achieve the purpose of foreground segmentation and ROI extraction. Among them, the segmentation algorithm based on maximum entropy threshold has the advantages of simple operation and stable performance, and is suitable for thermal image segmentation. Maximum entropy threshold segmentation was first proposed by Prasanna Sahoo et al. It selects the segmentation threshold by maximizing the entropy of the foreground and background. It has stronger robustness for processing images with complex or unevenly distributed information, and is widely used in industrial CT images and infrared thermal image analysis. Summary of the invention

[0007] The present invention proposes a key frame extraction method based on robust principal component analysis. Through this method, a key frame subsequence of the heat transfer process that is highly correlated with the defect characteristics and only contains the area of ​​interest in the heat map sequence can be extracted.

[0008] The technical solution of the present invention is as follows: first, the difference image between each adjacent image of the original thermal image sequence is calculated, the L1 norm of the difference result is obtained, and the starting frame of the pulse excitation is determined according to the maximum value; further, robust principal component analysis is used to decompose the thermal image sequence, and key frame extraction is realized according to the sparsity of the sparse matrix; finally, the block maximum entropy threshold segmentation method is used to extract the sample area in the thermal image.

[0009] A pulse thermal imaging sequence key frame extraction method based on robust principal component analysis, comprising:

[0010] (1) First, the difference image between each adjacent image in the original thermal image sequence is calculated, the L1 norm of the difference result is calculated, and the starting frame of the pulse excitation is determined according to the maximum value of the L1 norm;

[0011] (2) Then, robust principal component analysis is used to decompose the thermal image sequence to obtain the corresponding sparse matrix, and key frame extraction is achieved based on the sparsity of the sparse matrix;

[0012] (3) Finally, the local maximum entropy threshold segmentation method is used to extract the sample area in the thermal image and obtain a thermal image key frame sequence containing the sample area.

[0013] In practical application, the method of the present invention can be used to obtain a thermal image key frame sequence containing a sample area, and then the existing pulse thermal imaging detection method can be used to perform subsequent defect detection. Similarly, before performing the above method of the present invention, the original thermal image sequence of the object to be detected can be obtained using the existing pulse thermal imaging technology.

[0014] In step (1), for the original thermal image sequence Extract the image index N corresponding to the maximum L1 norm of the differential image of each adjacent frame s , the thermal image sequence after excitation is obtained as Where N t is the total number of frames in the original thermal image sequence. Sequence Γ0 contains N t Frame thermal image, each image size is N x ×N y .

[0015] Specifically, step (1) includes the following steps:

[0016] (1-1) Calculate the difference image of each adjacent frame in the sequence Γ0; Assume that any two adjacent thermal images I k-1 and I k , k∈{2,...,N t}, then its difference matrix D k =I k -I k-1 ;

[0017] (1-2) Calculate the matrix D k The L1 norm of ||D k ||1,||D k ||1=∑ i，j |D k (i, j)|; where i∈{1, 2, ..., N x}, j∈{1, 2, ..., N y};||*||1 means to find the L1 norm of *;

[0018] (1-3) Extract the maximum ||D k ||1 corresponds to the index N s , N s = argmax k (||D k ||1), according to the index value of the excitation start frame, the thermal image sequence after excitation is obtained

[0019] The thermal image sequence Γ1 after excitation contains both the dynamic heat transfer process and the thermal image in the heat balance state. Among them, the thermal image in the dynamic heat transfer process can reflect the defect information, so it is called the key thermal image. The key thermal image sequence Γ2 is further determined based on Γ1.

[0020] The present invention uses robust principal component analysis in thermal image sequence analysis. The RPCA algorithm models the image background as a low-rank component and separates the defect information into sparse components, thereby effectively decomposing the thermal image sequence and laying the foundation for judging the change process of the defect signal.

[0021] Specifically, in step (2), the specific method for extracting key frames is:

[0022] (2-1) Using robust principal component analysis to obtain the sparse matrix corresponding to the thermal image sequence after excitation;

[0023] (2-2) Then the sparsity of each vector in the sparse matrix is ​​calculated, and the sparsity is the quotient of the L1 norm of the corresponding vector and the size of the thermal image;

[0024] (2-3) Finally, a set of vectors with sparsity greater than a set value is obtained, and the corresponding thermal image is the key frame sequence.

[0025] In this step, Γ1 is expanded into a two-dimensional data matrix Robust principal component analysis is used to optimize the objective function min L，s (||L|| * +λ||S||1), decompose X into a low-rank matrix L and a sparse matrix S, that is, Subject to X=L+S. Among them, ||L|| * is the nuclear norm of the matrix L, and λ is a trade-off parameter used to adjust the optimization weights of the low-rank component and the sparse component.

[0026] Furthermore, the robust principal component analysis method mainly includes the following steps:

[0027] Step 2-1-1: Introduce the Lagrange multiplier matrix Y and the Lagrange multiplier scaling coefficient μ, and define T as the algorithm termination convergence threshold. Initialize the relevant parameters: where ||*||2 represents the L2 norm of *,

[0028]

[0029] Y=Y0 (6)

[0030] L=L0 (7)

[0031] S=S0 (8)

[0032] T=T0 (9)

[0033] Step 2-1-2: Perform singular value decomposition:

[0034]

[0035] U and V are orthogonal matrices consisting of left and right singular vectors, respectively, with dimensions N t -N s and N x ×N y .

[0036] Contract the singular value matrix ∑ to update the L matrix:

[0037]

[0038] Among them, σ i is the diagonal element of the i-th row of the singular value matrix ∑.

[0039] Step 2-1-3: Update the S matrix by soft threshold shrinkage:

[0040]

[0041] Step 2-1-4: Update the Y matrix based on the L matrix and the S matrix:

[0042] Y=Y+μ(XLS) (13)

[0043] Step 2-1-5: Update μ:

[0044] μ=ρμ (14)

[0045] Among them, ρ is a constant greater than 1, which is used to update μ to promote the convergence and optimization process.

[0046] Step 2-1-6: If (15) is satisfied, stop the iteration:

[0047] ||XLS|| F / ||X|| F <T (15)

[0048] Among them, ||*|| F is the Frobenius norm, which represents the square root of the sum of the squares of all elements in the matrix. If (15) is not satisfied, repeat steps 2-1-2 to 2-1-5 to update the matrix.

[0049] From the above steps, we can obtain the sparse matrix S corresponding to Γ1, which can be expressed as Among them, the vector s k (k∈{N s +1, ..., N t}) Corresponding thermal image I k A sparse vector of .

[0050] Then calculate each vector s k The sparsity(s k ),but

[0051] Finally, we get the vector set {s k1 ,s k2 , ..., s kn}, k1<k2<...<kn, and Finally, we get the key frame sequence Γ2 = {I k1 , I k2 , ..., I kn}.

[0052] In step (3), the local maximum entropy threshold segmentation method includes:

[0053] (3-1) Divide a certain frame of thermal image into multiple sub-areas;

[0054] (3-2) For each sub-region, a maximum entropy threshold segmentation method is used to obtain a sub-region segmentation mask, and multiple sub-region segmentation masks are spliced ​​to obtain a mask image with the same size as the original image;

[0055] (3-3) Multiply the obtained mask image with the extracted key frame sequence to obtain a thermal image key frame sequence containing the sample area.

[0056] Preferably, in step (3-1), the thermal image used is the first frame image I in the key frame sequence. k1 The number of sub-regions can be determined as needed, generally a×a 2 sub-regions, and a can be 2 to 5. For example, it can be divided into four sub-regions of 2×2, or nine sub-regions of 3×3, etc.

[0057] Taking the four sub-areas of 2×2 as an example, the following steps are included:

[0058] Step 3.1: I k1 Divide into four 2×2 sub-images I A1 , I A2 , I A3 , I A4 , each sub-image has a size of

[0059]

[0060] Step 3.2: For each sub-region I Ai, its grayscale range in Through linear transformation, its gray value range is mapped to Traversal Find a gray value by As the threshold, the sub-image is divided into two regions, the foreground and the background, and the entropy of the foreground and background regions is The sum is maximized, that is:

[0061]

[0062] Among them, the foreground and background entropy H fg,Ai (g i,k ) and H bg,Ai (g i,k )for:

[0063]

[0064] Among them, p(n) is the probability of gray level n, P fg,Ai (g i,k ) and P bg,Ai (g i,k ) are the cumulative probabilities of foreground and background respectively:

[0065]

[0066] h(n) represents the number of pixels with gray level n. According to (17)-(21), for the four sub-areas I A1 , I A2 , I A3 , I A4 Calculate the optimal grayscale threshold respectively

[0067] Step 3.3: Based on Get the segmentation mask image I′ of each sub-region respectively A1 , I′ A2 , I′ A3 , I′ A4 .

[0068]

[0069] The sub-region segmentation masks are concatenated to obtain a mask image I′ of the same size as the original image. A .

[0070]

[0071] Step 3.4: Mask I′ AThe corresponding elements are multiplied with the key frame sequence Γ1 extracted in step 2 in sequence to extract the sample area.

[0072]

[0073] Among them, ⊙ represents the multiplication of corresponding elements. Finally, the thermal image key frame sequence containing only the sample area is obtained.

[0074] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0075] The present invention can perfectly replace the existing manual key frame screening and ROI clipping. The solution makes full use of the spatiotemporal distribution characteristics of the thermal imaging sequence and aims to develop a complete and efficient pulse thermal imaging sequence key frame automatic extraction algorithm framework to improve the efficiency and automation level of defect recognition in practical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] Figure 1 The data processing flow chart of the present invention.

[0077] Figure 2 This is a diagram of the structure and geometry of the sample tested by the present invention.

[0078] Figure 3 It is the L1 norm trend diagram of the differential image of the present invention.

[0079] Figure 4 This is the key frame finally obtained by the present invention.

[0080] Figure 5 The segmentation process of the thermal image segmented by the present invention.

[0081] Figure 6 These are the overall segmentation results and local segmentation results of the thermal image segmented by the present invention.

[0082] Figure 7 This is the thermal image key frame sequence that only contains the sample area that is finally obtained by the present invention. DETAILED DESCRIPTION

[0083] To verify the present invention, polytetrafluoroethylene of different sizes and depths was inserted into the carbon fiber sheets at different depths and positions to simulate defects, and the samples were molded by molding. The sample structure and geometry were as follows: Figure 2 As shown. Pulse thermal imaging detection is performed on it, and key frame extraction and region of interest segmentation are performed on the acquired thermal image sequence. See the processing flow chart for Figure 1 , the method comprising:

[0084] Step 1: Obtain a thermal image sequence Γ0 = {I1, I2, ..., I 100}. The sampling time is 10 seconds, and it contains 100 frames of thermal images. The size of each image is 250×200.

[0085] Step 2: Determine the excitation start frame index N using the differential method s .

[0086] Step 2.1: Calculate the difference image and its L1 norm of each adjacent frame in the sequence Γ0. Assume that any two adjacent thermal images I k-1 and I k , k∈{2,...,100}, then its difference matrix D k :

[0087] D k =I k -I k-1 (1)

[0088] Calculate the matrix D k The L1 norm of ||D k ||1:

[0089] ||D k ||1=∑ i，j |D k (i, j)| (2)

[0090] Among them, i∈{1, 2, ..., 250}, j∈{1, 2, ..., 200}. We get ||D k ||1 Curve such as Figure 3 shown.

[0091] Step 2.2: Extract the maximum ||D k ||1 corresponds to the index N s .

[0092] N s = argmax k (||D k ||1)=33 (3)

[0093] That is, the 33rd frame is the excitation start frame. According to the index value of the excitation start frame, the thermal image sequence after excitation is obtained as Γ1={I 34 , I 35 , ..., I 00}.

[0094] Step 3: Sequence Γ1 contains thermal images of both the dynamic heat transfer process and the thermal equilibrium state. The thermal images of the dynamic heat transfer process can reflect the defect information, so they are called key thermal images. Based on Γ1, the key thermal image sequence Γ2 is further determined.

[0095] Expand Γ1 into a two-dimensional data matrix Robust principal component analysis is used to decompose X into a low-rank matrix L and a sparse matrix S by optimizing the objective function (4).

[0096] min L，S (||L|| * +λ||S||1) (4)

[0097] Subject to X=L+S

[0098] Among them, ||L|| * is the nuclear norm of the matrix L, λ is a trade-off parameter used to adjust the optimization weights of the low-rank component and the sparse component, min L，S (*) refers to obtaining the matrix L and the matrix S so that the function * value is minimized.

[0099] Step 3.1: Introduce the Lagrange multiplier matrix Y and the Lagrange multiplier scaling coefficient μ, and define T as the algorithm termination convergence threshold. Initialize the relevant parameters:

[0100]

[0101] Y=O (7)

[0102] L=0 (8)

[0103] S=O (9)

[0104] T=0.0001 (10)

[0105] Step 3.2: Perform singular value decomposition:

[0106]

[0107] U and V are orthogonal matrices composed of left and right singular vectors, respectively, with dimensions of 67×67 and 50000×50000.

[0108] Contract the singular value matrix ∑ to update the L matrix:

[0109]

[0110] Among them, σ i is the diagonal element of the i-th row of the singular value matrix ∑.

[0111] Step 3.3: Update the S matrix by soft threshold shrinkage:

[0112]

[0113] Step 3.4: Update the Y matrix based on the L matrix and the S matrix:

[0114] Y=Y+μ(XLS) (14)

[0115] Step 3.5: Update μ:

[0116] μ=1.1μ (15)

[0117] Step 3.6: If (16) is satisfied, stop the iteration:

[0118] ||XLS|| F / ||X|| F <T (16)

[0119] Among them, ||*|| F represents the Frobenius norm. If (16) is not satisfied, repeat steps 3.2-3.5 to update the matrix. Obtain the sparse matrix S corresponding to the sequence Γ1 through steps 3.1-3.6;

[0120] Step 3.7: The sparse matrix can be expressed as S = [s 34 ,s 35 , ..., s 100 ]. Among them, the vector s k (k∈{34,...,100}) corresponds to the thermal image I k Calculate the sparse vector of each vector s k The sparsity(s k ),but

[0121]

[0122] Step 3.8: Get the top 10% vector set with the largest sparsity {s 34 ,s 35 ,s 36 ,s 37 ,s 38 Finally, the key thermal image sequence Γ2 = {I 34 , I 35 , I 36 , I 37 , I 38}. Figure 4 is the obtained key frame sequence.

[0123] Step 4: Select the first frame image I in the sequence Γ234 ,The local maximum entropy threshold segmentation algorithm is used to extract the sample area contained in the thermal image.

[0124] Step 4.1: I 34 Divide into four 2×2 sub-images I A1 , I A2 , I A3 , I A4 , each sub-image size is 125 × 100. The splitting process is as follows Figure 5 shown.

[0125]

[0126] Step 4.2: For each sub-region I Ai , its grayscale range in Through linear transformation, its gray value range is mapped to Traversal Find a gray value by As the threshold, the thermal image is divided into two areas, the foreground and the background, and the entropy of the foreground and background areas is Maximize the sum.

[0127]

[0128] Foreground and background entropy H fg,Ai (g i,k ) and H bg,Ai (g i,k )for:

[0129]

[0130] Among them, p(n) is the probability of gray level n, P fg,Ai (g i,k ) and P bg,Ai (g i,k ) are the cumulative probabilities of foreground and background respectively:

[0131]

[0132] h(n) represents the number of pixels with grayscale n.

[0133] Finally, four sub-areas I are obtained A1 , I A2 , I A3 , I A4 The optimal grayscale threshold is The corresponding entropy is:

[0134]

[0135] Step 4.3: Based on Get the segmentation mask I′ of each sub-region respectively A1 , I′ A2 , I′ A3 , I′ A4 .

[0136]

[0137] The sub-region segmentation masks are concatenated to obtain a mask image I′ of the same size as the original image. A . Figure 6 I′ A1 , I′ A2 , I′ A3 , I′ A4 and I′ A .

[0138]

[0139] Step 4.4: Mask I′ A The corresponding elements are multiplied with the key frame sequence Γ2 extracted in step 3 in sequence to extract the sample area.

[0140]

[0141] Finally, a sequence of thermal image key frames containing only the sample area is obtained. Figure 7 is the final thermal image key frame sequence that only contains the sample area.

Claims

1. A pulse thermal imaging sequence key frame extraction method based on robust principal component analysis, characterized in that: include: (1) First, the difference image between each adjacent image in the original thermal image sequence is calculated, the L1 norm of the difference result is calculated, and the starting frame of the pulse excitation is determined according to the maximum value of the L1 norm; (2) Then, robust principal component analysis is used to decompose the thermal image sequence to obtain the corresponding sparse matrix, and key frame extraction is achieved based on the sparsity of the sparse matrix; (3) Finally, the local maximum entropy threshold segmentation method is used to extract the sample area in the thermal image and obtain a thermal image key frame sequence containing the sample area.

2. The method for extracting key frames from pulse thermal imaging sequences based on robust principal component analysis according to claim 1, characterized in that: In step (1), for the original thermal image sequence, the image index N corresponding to the maximum L1 norm of the differential image of each adjacent frame is extracted. s , obtain the thermal image sequence after excitation.

3. The method for extracting key frames from pulse thermal imaging sequences based on robust principal component analysis according to claim 2, characterized in that: The index starting number of the thermal image sequence after excitation is N s +1.

4. The method for extracting key frames from pulse thermal imaging sequences based on robust principal component analysis according to claim 1, characterized in that: In step (2), the specific method of key frame extraction is: (2-1) A robust principal component analysis algorithm is used to obtain a sparse matrix corresponding to the thermal image sequence after excitation; (2-2) Then the sparsity of each vector in the sparse matrix is ​​calculated, and the sparsity is the quotient of the L1 norm of the corresponding vector and the size of the thermal image; (2-3) Finally, a set of vectors with sparsity greater than a set value is obtained, and the corresponding thermal image is the key frame sequence.

5. The method for extracting key frames from pulse thermal imaging sequences based on robust principal component analysis according to claim 4, characterized in that: In step (2-3), the first 5 to 20% of the vector sets with the largest sparsity are obtained as the vector set.

6. The method for extracting key frames from pulse thermal imaging sequences based on robust principal component analysis according to claim 1, characterized in that: In step (3), the local maximum entropy threshold segmentation method includes: (3-1) dividing a certain frame of thermal image into a plurality of sub-regions; (3-2) For each sub-region, a maximum entropy threshold segmentation method is used to obtain a sub-region segmentation mask, and multiple sub-region segmentation masks are spliced ​​to obtain a mask image with the same size as the original image; (3-3) Multiply the obtained mask image with the extracted key frame sequence to obtain a thermal image key frame sequence containing the sample area.

7. The method for extracting key frames from pulse thermal imaging sequences based on robust principal component analysis according to claim 6, characterized in that: In step (3-1), the selected thermal image is the first frame image in the key frame sequence.

Images