Deep neural network-based defect detection system and defect detection method using composite thermographic data analysis

Abstract

The present invention relates to a deep neural network-based defect detection system and defect detection method using composite thermographic data analysis, the system and method making it possible to: design the structure of a deep neural network model capable of being trained in consideration of the time-series characteristics of composite thermographic data obtained by a thermal imaging camera; and, by using same, accurately detect fine-sized defects and internal defects without disassembling a composite structure.

Description

Defect detection system and method using composite thermal image data analysis based on deep neural networks

[0001] The present invention relates to a defect detection system and a defect detection method using a deep neural network-based composite thermal image data analysis, and more specifically, to a defect detection system and a defect detection method using a deep neural network-based composite thermal image data analysis, which design a structure of a deep neural network model capable of learning by considering the time-series characteristics of thermal image data of a composite acquired by a thermal image camera, and utilize the same to accurately detect microscopic defects and internal defects without disassembling a composite material structure.

[0002] In general, when performing management and maintenance of composite structures in operation, defect detection of composite materials is performed by applying various contact or non-contact methods and developed technologies.

[0003] Among the existing on-site defect detection methods for composite materials, visual inspection can only identify external defects. Vibration / displacement techniques have low resolution, and AE techniques suffer from noise issues. Furthermore, laser UT technology faces the challenge of requiring composite structures to be disassembled and transported to a location where inspection is possible to identify defects during operation.

[0004] Thermal imaging, one of the techniques that can overcome these problems, has the advantage of being able to detect internal defects and, because it is non-contact, does not require disassembly of the structure.

[0005] However, the thermal imaging technique currently in use has limitations in that it is difficult to detect defects that are smaller in size and farther from the heat source. Therefore, there is a need for the development of a technology that can detect small defects and defects inside composite materials without disassembling the structure during structural operation using a thermal imaging technique based on a deep neural network.

[0006] The present invention is intended to solve the above-mentioned problems, and designs a structure of a deep neural network model capable of learning by considering the time-series characteristics of thermal image data of a composite material acquired by a thermal imaging camera, and utilizes the same to provide a defect detection system and a defect detection method using composite thermal image data analysis based on a deep neural network that can accurately detect microscopic defects and internal defects without disassembling the composite material structure.

[0007] A defect detection system (100) using composite thermal image data analysis based on a deep neural network according to one embodiment of the present invention may include a composite thermal image data learning unit (110) that learns composite thermal image data to generate a defect detection model, a composite thermal image data feature extraction unit (120) that extracts features of thermal image data for an input composite to be inspected using the defect detection model, and a composite thermal image data defect determination unit (130) that determines whether the composite to be inspected has a defect based on the extracted features.

[0008] In one embodiment, the composite thermal image data learning unit (110) can learn composite thermal image data based on a deep neural network including a partial thermal gradient feature extraction deep neural network, a thermal gradient time series feature extraction deep neural network, and a global thermal gradient feature extraction deep neural network.

[0009] In one embodiment, the composite thermal image data learning unit (110) may generate a plurality of non-overlapping segmented areas of a specific size from the thermal image data for learning, then extract a specific segmented area and input it into the partial thermal gradient feature extraction deep neural network so that a plurality of feature maps are extracted, and then input the plurality of extracted feature maps into the thermal gradient time-series feature extraction deep neural network.

[0010] In one embodiment, the composite thermal image data learning unit (110) divides a specific segmented area input to the partial thermal gradient feature extraction deep neural network into a plurality of patches and then inputs the patches, thereby learning the interrelationship of temperature gradients between the plurality of patches through the partial thermal gradient feature extraction deep neural network.

[0011] In one embodiment, when a plurality of feature maps are input to the thermal gradient time-series feature extraction deep neural network, the composite thermal image data learning unit (110) learns the time-series correlation for temperature changes over a specific period of time for the same segmented area, and performs feature extraction for a specific period of time for each of a plurality of non-overlapping segmented areas to extract a plurality of feature maps, which are then input to the global thermal gradient feature extraction deep neural network.

[0012] In one embodiment, when a plurality of feature maps are input to the global thermal gradient feature extraction deep neural network, the composite thermal image data learning unit (110) can learn a global correlation between the time series characteristics of the thermal gradient of each segmented area for the plurality of input feature maps and the entire size of the training thermal image data.

[0013] In one embodiment, the composite thermal image data learning unit (110) learns global correlations for the entire size of the training thermal image data through the global thermal gradient feature extraction deep neural network, and converts the entire dataset length of the training thermal image data into time, and repeats the learning for the corresponding time, thereby completing the learning of the defect detection model.

[0014] In one embodiment, the composite thermal image data feature extraction unit (120) can extract a feature map of thermal image data for the input composite material to be inspected using the defect detection model for which learning has been completed.

[0015]

[0016] A defect detection method using composite thermal image data analysis based on a deep neural network according to another embodiment of the present invention may include a step of learning composite thermal image data in a composite thermal image data learning unit to create a defect detection model, a step of extracting features of thermal image data for an input composite material to be inspected using the defect detection model in a composite thermal image data feature extraction unit, and a step of determining whether the composite material to be inspected is defective based on the extracted features in a composite thermal image data defect determination unit.

[0017] In one embodiment, the step of learning the composite thermal image data to create a defect detection model comprises: a step of: generating, in the composite thermal image data learning unit, a plurality of non-overlapping segmented areas of a specific size from the training thermal image data, extracting a specific segmented area and inputting the extracted feature maps into a partial thermal gradient feature extraction deep neural network; and inputting the extracted feature maps into a thermal gradient time-series feature extraction deep neural network; a step of: learning a time-series correlation for a temperature change over a specific time period for the same segmented area; performing feature extraction for a specific time period for each of the plurality of non-overlapping segmented areas; and inputting the extracted feature maps into a global thermal gradient feature extraction deep neural network; and a step of: inputting, in the composite thermal image data learning unit, a step of: extracting a plurality of feature maps by inputting the plurality of feature maps into a global thermal gradient feature extraction deep neural network; and a step of: inputting a ... A step may be included to learn a global correlation of the time-series characteristics of the thermal gradient for the entire size of the above-mentioned training thermal image data.

[0018] In one embodiment, the step of inputting the extracted plurality of feature maps into a thermal gradient time-series feature extraction deep neural network may include a step of, in the composite thermal image data learning unit, dividing a specific segmented area input into a plurality of patches and then inputting the same into a plurality of patches, so that the partial thermal gradient feature extraction deep neural network learns the correlation between temperature gradients between the plurality of patches.

[0019] In one embodiment, the step of learning a global correlation for the time-series characteristics of the thermal gradient of each segmented area for the plurality of input feature maps for the entire size of the training thermal image data may include a step of learning, in the composite thermal image data learning unit, a global correlation for the training thermal image data for the entire size through the global thermal gradient feature extraction deep neural network, converting the entire dataset length of the training thermal image data into time, and repeating the learning for the corresponding time, so that the learning of the defect detection model is completed.

[0020] According to the present invention, there is an advantage in that microscopic defects and internal defects can be accurately detected without disassembling a composite material structure.

[0021] In particular, according to the present invention, when a person directly detects a defect using an existing thermal imaging technique, the smaller the size of the defect and the farther away it is from the heat source, the more difficult it is to detect the defect using the existing technique, which has the advantage of solving the technical difficulty.

[0022] In addition, according to the present invention, by being used in a device that can automatically perform defect detection of a composite material, it has the advantage of being able to automatically detect defects without human intervention, thereby drastically reducing the cost and time required for defect detection.

[0023] FIG. 1 is a diagram showing the configuration of a defect detection system (100) using composite thermal image data analysis based on a deep neural network according to one embodiment of the present invention.

[0024] Figure 2 is a flowchart showing the entire process of learning and completing learning of a defect detection model by learning learning thermal image data in a composite thermal image data learning unit (110).

[0025] FIG. 3 is a diagram for explaining a segmented area input to a partial thermal gradient feature extraction deep neural network according to one embodiment of the present invention.

[0026] Figure 4 is a drawing for explaining the concept of a deep neural network for extracting partial thermal gradient features of a composite thermal image data learning unit (110).

[0027] Figure 5 is a diagram for explaining the concept of a deep neural network for extracting thermal gradient time series features of a composite thermal image data learning unit (110).

[0028] FIG. 6 is a drawing for explaining an embodiment in which the divided areas illustrated in FIG. 3 can be divided without overlapping.

[0029] FIG. 7 is a drawing for explaining another embodiment in which the divided areas shown in FIG. 3 can be divided without overlapping.

[0030] FIG. 8 is a drawing for explaining another embodiment in which the divided areas shown in FIG. 3 can be divided without overlapping.

[0031] FIG. 9 is a drawing for explaining another embodiment in which the divided areas shown in FIG. 3 can be divided without overlapping.

[0032] Figure 10 is a drawing for explaining the concept of a deep neural network for extracting global thermal gradient features of a composite thermal image data learning unit (110).

[0033]

[0034] <Explanation of symbols>

[0035] 100: Defect Detection System Using Composite Thermal Image Data Analysis Based on Deep Neural Networks

[0036] 110: Composite thermal image data learning unit

[0037] 120: Composite thermal image data feature extraction unit

[0038] 130: Composite thermal image data defect determination unit

[0039] Hereinafter, embodiments of the present invention will be described in detail with reference to the attached drawings so that those skilled in the art can easily implement the present invention. The present invention may be implemented in various different forms and is not limited to the embodiments described herein.

[0040] In order to clearly explain the present invention, parts that are not related to the description are omitted, and the same reference numerals are used for identical or similar components throughout the specification.

[0041] In addition, in various embodiments, components having the same configuration are described only in representative embodiments using the same symbols, and in other embodiments, only configurations different from the representative embodiments are described.

[0042] Throughout the specification, when a part is said to be "connected" to another part, this includes not only "directly connected" but also "indirectly connected" with other elements intervening. Furthermore, when a part is said to "include" a component, this may mean that the other component is included, rather than excluded, unless otherwise specifically stated.

[0043]

[0044] FIG. 1 is a diagram showing the configuration of a defect detection system (100) using composite thermal image data analysis based on a deep neural network according to one embodiment of the present invention.

[0045] Referring to FIG. 1, a defect detection system (100) using a deep neural network-based composite thermal image data analysis according to one embodiment of the present invention can be largely composed of a composite thermal image data learning unit (110), a composite thermal image data feature extraction unit (120), and a composite thermal image data defect determination unit (130).

[0046] First, the composite thermal image data learning unit (110) learns composite thermal image data to generate a defect detection model. At this time, the learning thermal image data is learned based on a deep neural network including a partial thermal gradient feature extraction deep neural network, a thermal gradient time-series feature extraction deep neural network, and a global thermal gradient feature extraction deep neural network.

[0047] The process of learning thermal image data for training in the composite thermal image data learning unit (110) and creating a defect detection model is as follows.

[0048] FIG. 2 is a flowchart sequentially showing the entire process of learning thermal image data for learning in a composite material thermal image data learning unit (110) and performing and completing learning of a defect detection model, FIG. 3 is a diagram for explaining a segmented area input to a partial thermal gradient feature extraction deep neural network according to an embodiment of the present invention, FIG. 4 is a diagram for explaining the concept of a partial thermal gradient feature extraction deep neural network of a composite material thermal image data learning unit (110), FIG. 5 is a diagram for explaining the concept of a thermal gradient time-series feature extraction deep neural network of a composite material thermal image data learning unit (110), FIG. 6 is a diagram for explaining an embodiment in which the segmented area shown in FIG. 3 can be divided without overlapping, FIG. 7 is a diagram for explaining another embodiment in which the segmented area shown in FIG. 3 can be divided without overlapping, FIG. 8 is a diagram for explaining another embodiment in which the segmented area shown in FIG. 3 can be divided without overlapping, and FIG. 9 is a diagram for explaining another embodiment in which the segmented area shown in FIG. 3 can be divided without overlapping. It is a drawing, and FIG. 10 is a drawing for explaining the concept of a deep neural network for extracting global thermal gradient features of a composite thermal image data learning unit (110).

[0049] Looking at FIGS. 2 to 10, the composite thermal image data learning unit (110) is composed of a deep neural network including a partial thermal gradient feature extraction deep neural network, a thermal gradient time series feature extraction deep neural network, and a global thermal gradient feature extraction deep neural network.

[0050] Referring to FIGS. 2 to 4, the composite thermal image data learning unit (110) extracts a 64*64 sized segmented area from the 256*256 sized training thermal image data and inputs it into the partial thermal gradient feature extraction deep neural network. At this time, a total of 49 64*64 sized segmented areas can be generated from the entire 256*256 sized training thermal image data without overlapping each other.

[0051] At this time, the arrangement of the 64*64 sized segmented areas that are generated without overlapping each other from the entire learning thermal image data can be arranged in various ways as shown in Figs. 6 to 9.

[0052] Meanwhile, the composite thermal image data learning unit (110) divides each segmented area of ​​64*64 size into a total of 16 patches and inputs them into the partial thermal gradient feature extraction deep neural network during the process of inputting them. Through this, the partial feature extraction deep neural network can learn the interrelationship of the temperature gradients between the 16 patches and extract a feature map accordingly. This will be examined in more detail as follows.

[0053] Assuming that the 256×256 sized training thermal image data input into the composite thermal image data learning unit (110) has a total length of m, the composite thermal image data learning unit (110) reflects m corresponding to the total length as time t and extracts data for each t. Each of the m extracted 256×256 sized data is divided into a 64×64 sized segment area. At this time, the size of each segment area is not divided equally as shown in FIGS. 6 to 9.

[0054] Looking at Figure 4, at time t, which is the extraction point of each partition of size 64*64, the i-th partition is P t,i It can be expressed as . If, when time is t, the composite thermal image data learning unit (110) is the i-th segmented area P t,i It is divided into 16 patches with a size of 16*16. The partial thermal gradient feature extraction deep neural network inputs this into the thermal gradient feature extraction deep neural network. If the time to be analyzed is n, the partial thermal gradient feature extraction deep neural network inputs the i-th partition area P divided into 16 patches. t,iThe process of inputting the thermal gradient feature extraction deep neural network is repeated until the time becomes from t to t+n. By this iterative process, P t,i From P t+n,i A total of n feature maps are extracted.

[0055] The composite thermal image data learning unit (110) inputs the total n feature maps extracted in this way into the thermal gradient time series feature extraction neural network. At this time, as shown in Fig. 5, the composite thermal image data learning unit (110) also performs this process in the i-th segmentation area, P t,i This P i,49 This process is repeated until i+1 is reached. By repeating this process, P t,1 From P t,49 A total of 49 feature maps can be extracted.

[0056] The composite thermal image data learning unit (110) inputs the 49 feature maps extracted in this way into the global thermal gradient feature extraction deep neural network. At this time, as shown in FIG. 10, the composite thermal image data learning unit (110) also performs this process in the same manner as P t,i This P m,i This will be repeated until it becomes .

[0057] The composite thermal image data learning unit (110) ends learning for the corresponding defect detection model when all analysis of the thermal image data for learning is completed from t=0 to t=m.

[0058] When the learning of the composite thermal image data learning unit (110) is completed, the learning parameters of the partial thermal gradient feature extraction deep neural network, the thermal gradient feature extraction deep neural network, and the global thermal gradient feature extraction deep neural network are fixed, and the partial thermal gradient feature extraction deep neural network, the thermal gradient feature extraction deep neural network, and the global thermal gradient feature extraction deep neural network of the composite thermal image data feature extraction unit (120) described below are completed.

[0059]

[0060] The composite thermal image data feature extraction unit (120) uses the defect detection model thus completed to extract a feature map from the thermal image data for the composite material to be inspected with a size of 256*256, without changing the learning parameters, in the same way as the composite thermal image data learning unit (110) previously performed.

[0061] The feature map extracted through the composite thermal image data feature extraction unit (120) is input to the composite thermal image data defect judgment unit (130), and the composite thermal image data defect judgment unit (130) analyzes m / n feature maps for the entire thermal image data for the composite material to be inspected, with the extracted time being up to t=m seconds, to judge and classify whether there is a defect (Defect) or normal (Non-Defect).

[0062]

[0063] Next, we will examine in order the entire process of learning composite thermal image data and determining whether or not the composite material to be inspected has a defect using the defect detection system (100) that utilizes the composite thermal image data analysis based on the deep neural network examined above.

[0064] First, the composite thermal image data learning unit (110) learns the composite thermal image data to generate a defect detection model. In this process, the composite thermal image data learning unit (110) extracts a 64*64 sized segmented area from the 256*256 sized training thermal image data and inputs it to a partial thermal gradient feature extraction deep neural network, dividing each segmented area into a total of 16 patches and inputting them. Through this, the partial feature extraction deep neural network can learn the interrelationship of the temperature gradients between the 16 patches and extract a feature map accordingly.

[0065] Next, the composite thermal image data learning unit (110) inputs the extracted multiple feature maps into a thermal gradient time series feature extraction neural network, and repeats this process to extract multiple feature maps. The extracted multiple feature maps are then input into a global thermal gradient feature extraction deep neural network, and this process is similarly repeated multiple times. Here, multiple times may mean until the entire dataset of thermal image data for training is learned.

[0066] Next, the composite thermal image data learning unit (110) ends learning for the corresponding defect detection model when all analysis of the thermal image data for learning is completed from t=0 to t=m.

[0067] After learning is completed, the learning parameters of the partial thermal gradient feature extraction deep neural network, the thermal gradient feature extraction deep neural network, and the global thermal gradient feature extraction deep neural network are fixed, and the partial thermal gradient feature extraction deep neural network, the thermal gradient feature extraction deep neural network, and the global thermal gradient feature extraction deep neural network of the composite thermal image data feature extraction unit (120) are completed.

[0068] After this, when thermal image data of 256*256 size for the composite material to be inspected is input, the composite thermal image data feature extraction unit (120) extracts a feature map without changing the learning parameters in the same manner as the composite thermal image data learning unit (110) previously performed, and inputs the extracted feature map to the composite thermal image data defect judgment unit (130).

[0069] The composite thermal image data defect judgment unit (130) analyzes m / n feature maps extracted from the entire thermal image data for the composite material to be inspected up to a time of t=m seconds, and judges and classifies whether there is a defect (Defect) or normal (Non-Defect).

[0070]

[0071] Although the present invention has been described above with reference to preferred embodiments thereof, it will be understood by those skilled in the art that various modifications and changes may be made to the present invention without departing from the spirit and scope of the present invention as set forth in the claims below.

[0072] The present invention is a technology that can accurately detect microscopic defects and internal defects without disassembling a composite material structure, and can be widely used in the non-destructive testing industry to realize its practical and economic value.

Claims

1. A composite thermal image data learning unit (110) that learns composite thermal image data to create a defect detection model; A composite thermal image data feature extraction unit (120) that extracts features of thermal image data for an input inspection target composite material using the above defect detection model; and A composite thermal image data defect determination unit (130) that determines whether the composite material to be inspected has a defect based on the extracted features; characterized in that it includes; Defect detection system using composite thermal image data analysis based on deep neural network.

2. In paragraph 1, The above composite thermal image data learning unit (110) is A method for learning composite thermal image data based on a deep neural network including a partial thermal gradient feature extraction deep neural network, a thermal gradient time-series feature extraction deep neural network, and a global thermal gradient feature extraction deep neural network. Defect detection system using composite thermal image data analysis based on deep neural network.

3. In paragraph 2, The above composite thermal image data learning unit (110) is A method for generating a plurality of non-overlapping segmented areas of a specific size from thermal image data for learning, extracting a specific segmented area and inputting it into the partial thermal gradient feature extraction deep neural network so that a plurality of feature maps are extracted, and then inputting the extracted plurality of feature maps into the thermal gradient time-series feature extraction deep neural network. Defect detection system using composite thermal image data analysis based on deep neural network.

4. In paragraph 3, The above composite thermal image data learning unit (110) is A method characterized in that a specific segmented area input to the partial thermal gradient feature extraction deep neural network is divided into a plurality of patches and then input, so that the partial thermal gradient feature extraction deep neural network learns the correlation between the temperature gradients between the plurality of patches. Defect detection system using composite thermal image data analysis based on deep neural network.

5. In paragraph 3, The above composite thermal image data learning unit (110) is In the case where multiple feature maps are input to the above thermal gradient time-series feature extraction deep neural network, the time-series correlation for temperature changes during a specific time period for the same segmented area is learned, and feature extraction is performed for each of multiple non-overlapping segmented areas during a specific time period to extract multiple feature maps, which are then input to the global thermal gradient feature extraction deep neural network. Defect detection system using composite thermal image data analysis based on deep neural network.

6. In paragraph 3, The above composite thermal image data learning unit (110) is When a plurality of feature maps are input to the global thermal gradient feature extraction deep neural network, the time series features of the thermal gradient of each segmented area for the plurality of input feature maps are learned to have a global correlation with respect to the entire size of the training thermal image data. Defect detection system using composite thermal image data analysis based on deep neural network.

7. In paragraph 6, The above composite thermal image data learning unit (110) is The global thermal gradient feature extraction deep neural network is used to learn global correlations for the full-size training thermal image data. The entire data set length of the above learning thermal image data is converted into time, and learning is repeated for the corresponding time, so that the learning of the defect detection model is completed. Defect detection system using composite thermal image data analysis based on deep neural network.

8. In paragraph 7, The above composite thermal image data feature extraction unit (120) is A method characterized in that a feature map of thermal image data for an input inspection target composite material is extracted using the above learning-completed defect detection model. Defect detection system using composite thermal image data analysis based on deep neural network.

9. In the composite thermal image data learning unit, a step of learning composite thermal image data to create a defect detection model; In the composite thermal image data feature extraction unit, a step of extracting features of thermal image data for the input inspection target composite material using the above defect detection model; and A composite thermal image data defect judgment unit characterized by including a step of judging whether the composite material to be inspected is defective based on the extracted features; Defect detection method using composite thermal image data analysis based on deep neural network.

10. In paragraph 9, The step of learning the above composite thermal image data to create a defect detection model is as follows. In the above composite thermal image data learning unit, a step of generating a plurality of non-overlapping segmented areas of a specific size from the thermal image data for learning, extracting a specific segmented area and inputting it into a partial thermal gradient feature extraction deep neural network so that a plurality of feature maps are extracted, and then inputting the plurality of extracted feature maps into a thermal gradient time-series feature extraction deep neural network; In the above composite thermal image data learning unit, when a plurality of feature maps are input to the thermal gradient time-series feature extraction deep neural network, a step of learning the time-series correlation for temperature changes for a specific time period for the same segmented area, and performing feature extraction for a specific time period for each of a plurality of non-overlapping segmented areas to extract a plurality of feature maps, and then inputting them to the global thermal gradient feature extraction deep neural network; and In the above composite thermal image data learning unit, when a plurality of feature maps are input to the global thermal gradient feature extraction deep neural network, a step is included to learn the global correlation of the time series features of the thermal gradient of each divided area for the plurality of input feature maps for the entire size of the learning thermal image data. Defect detection method using composite thermal image data analysis based on deep neural network.

11. In paragraph 9, The step of inputting the above extracted multiple feature maps into a thermal gradient time series feature extraction deep neural network is as follows. In the composite thermal image data learning unit, a step is included in which a specific segmented area input to the partial thermal gradient feature extraction deep neural network is divided into a plurality of patches and then input, so that the temperature gradients between the plurality of patches are learned through the partial thermal gradient feature extraction deep neural network. Defect detection method using composite thermal image data analysis based on deep neural network.

12. In paragraph 10, The step of learning the global correlation of the time-series characteristics of the thermal gradient of each segmented area for the multiple feature maps input above for the entire size of the training thermal image data is as follows. In the above composite thermal image data learning unit, a step is included in which the global correlation for the entire size of the training thermal image data is learned through the global thermal gradient feature extraction deep neural network, and the entire dataset length of the training thermal image data is converted into time, and then the learning is repeated for the corresponding time, so that the learning of the defect detection model is completed. Defect detection method using composite thermal image data analysis based on deep neural network.

Images